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Abstract 


As  current  United  States  Department  of  Defense  (DoD)  system  development  and 
engineering  activities  continue  to  be  challenged  by  formulation  of  larger  and  more 
complex  systems,  DoD’s  methods,  processes,  and  tools  (MPT)  for  effectively  and 
efficiently  addressing  these  challenges  are  likewise  being  challenged.  The  goal  of  this 
research  was  to  develop  a  mixed  methodological  approach  to  examine  systems 
development  maturity.  Qualitatively  we  intended  to  uncover  and  investigate  the  key 
characteristics  that  drive  the  development  of  large  scale,  complex  systems. 
Quantitatively  we  used  these  key  characteristics  to  formulate  a  collection  of  analytical 
MPT  to  assist  in  making  informed  systems  engineering  management  decisions.  To 
advance  the  state  of  practice  of  this  research,  all  MPT  developed  under  this  task  were 
validated  through  application  on  designated  projects.  The  validation  effort  was 
designed  to  determine  if  they  could  be  effectively  implemented  as  a  best  practice  across 
the  Department  of  Defense.  The  need  for  this  research  is  precipitated  by  the  need  for 
system  engineering,  development,  and  cost  models  that  adequately  incorporate  the 
unique  aspects  of  system  and  technology  insertion  and  integration.  This  was  then 
predicated  on  the  following  task  objectives: 

•  Leverage  prior  investments  made  in  the  System  Readiness  Level  (SRL)  body  of 
knowledge  to  explore  the  effects  of  technology  and  integration  maturity  on 
systems  engineering  effort  and  cost; 

•  Expand  the  scope  and  applicability  of  the  SRL  to  address  potential  systems 
engineering  MPT;  and 

•  Enhance  current  SRL  methods  and  tools  to  incorporate  research-derived 
insights,  provide  expanded  functionality,  and  demonstrate  the  utility  of  the  tools 
in  the  context  of  a  pilot  project. 

At  present,  the  SRL  is  a  descriptive  model  that  characterizes  the  effects  of  technology 
and  integration  maturity  on  a  system  engineering  effort,  particularly  with  respect  to 
integrating  discrete  functional  systems  into  a  coherent  mission  capability.  The  SRL 
model,  as  developed  by  the  Systems  Development  &  Maturity  Laboratory  (SysDML)  at 
Stevens  Institute  of  Technology,  has  been  validated  by  the  NAVSEA  PMS  420,  Littoral 
Combat  Ship  Mission  Module  Program  Office  in  collaboration  with  Northrop  Grumman 
Corporation  which  used  it  to  monitor  development  and  integration  progress  from  a 
system  perspective.  The  SRL  and  supporting  assessment  methodology  has  proven  itself 
to  be  a  promising  mechanism  for  understanding  the  effects  of  technology  and 
integration  maturity  in  a  systems  engineering  context.  This  task  extends  the  research  to 
better  characterize  the  drivers  of  integration  effort,  and  to  explore  application  of  the 
SRL  to  broader  areas  of  the  systems  engineering  management,  where  validated  models 
and  supporting  tools  are  lacking. 
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1  Summary 


At  present,  the  System  Readiness  Level  (SRL)  developed  by  the  Systems  Development  & 
Maturity  Laboratory  (SysDML)  at  Stevens  Institute  of  Technology  is  a  descriptive  model 
that  characterizes  the  effects  of  technology  and  integration  maturity  on  a  systems 
engineering  effort,  particularly  with  respect  to  integrating  discrete  functional  systems 
into  a  coherent  mission  capability.  This  research  expanded  upon  this  work  to  develop  a 
mixed  methodological  approach  to  examine  systems  development  maturity. 
Qualitatively  we  continue  to  uncover  and  investigate  the  key  characteristics  that  drive 
the  development  of  large  scale,  complex  systems.  Quantitatively  we  used  these  key 
characteristics  to  formulate  a  collection  of  analytical  methods,  processes,  and  tools 
(MPT)  to  assist  in  making  informed  systems  engineering  management  decisions. 

To  advance  the  state  of  practice  of  this  research,  the  MPT  developed  under  this  task 
were  validated  in  a  collaborative  research  effort  with  the  US  Navy  NAVSEA  PMS  420, 
Littoral  Combat  Ship  Mission  Module  Program  Office  and  Northrop  Grumman 
Corporation  on  the  US  Navy  NAVSEA  PMS  420,  Littoral  Combat  Ship  Mission  Module 
Program.  Through  this  validation,  the  SRL  and  supporting  assessment  methodology  as 
developed  in  this  research  have  proven  to  be  a  promising  mechanism  for  understanding 
the  effects  of  technology  and  integration  maturity  in  a  systems  engineering  context.  In 
addition,  the  MPT  research  and  development  under  this  task  have  demonstrated  utility 
for  defining  system  status  and  providing  leading  indicators  of  development  risks.  The 
success  of  the  SRL’s  implementation  thus  far  highlights  the  potential  benefits  of 
extending  the  research  to  better  characterize  the  drivers  of  integration  effort,  and  to 
explore  application  of  the  SRL  to  broader  areas  of  the  systems  engineering  domains, 
particularly  with  respect  to  large  scale,  complex  implementations,  where  validated 
models  and  supporting  tools  are  lacking. 

In  summary,  the  work  performed  in  this  task  by  the  SysDML  of  Stevens  Institute  of 
Technology  in  conjunction  with  Texas  A&M  University  was  partitioned  into  four  areas 
linking  SRL  to  development  architectures,  milestones,  and  costs: 

1.  Aligning  SRL  Methods  with  Architectures 

2.  Mapping  SRL/ITRL  to  Development  Activities/Status 

3.  Leveraging  SRL  Relationships  to  Allocate  Resources 

4.  TRL  and  IRL  Evaluation  Linkage  to  Performance  Measures 
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2  Introduction 


Current  United  States  (US)  Department  of  Defense  (DoD)  system  development  and 
engineering  activities  continue  to  be  challenged  by  formulation  of  larger  and  more 
complex  systems  or  systems  of  systems  (SoS).  In  many  ways,  these  development 
paradigms  are  contesting  many  of  the  engineering,  management,  and  acquisition 
practices  that  have  been  used  for  decades  in  the  development  of  stand-alone  systems. 
Similarly,  complex  system  development  has  made  the  management  of  systems 
engineering  activities  more  difficult  to  monitor  and  control  due  to  the  exponential 
growth  of  technologies  and  integrations  being  incorporated  under  a  common  effort. 
Thus,  there  is  a  growing  need  for  more  systematic  and  systemic  approaches  to 
monitoring  development  and  integration  of  systems.  This  necessitates  the  development 
of  new  methods,  processes,  and  tools  (MPT)  through  best  practices  in  order  to  govern 
the  many  unique  aspects  of  systems  development  and  acquisition  programs,  and  be  able 
to  compare  actual  progress  against  planned  accomplishment  from  a  technical 
perspective.  In  a  continued  effort  to  address  some  of  these  issues,  this  report 
summarizes  the  research  performed  under  this  task  by  the  Systems  Development  & 
Maturity  Laboratory  (SysDML)  at  Stevens  Institute  of  Technology  in  system  maturity 
assessment  for  developmental  lifecycles. 


2.1  Rationale  for  Development  of  System  Maturity  Assessment 

In  1999,  the  US  General  Accounting  Office  (GAO)  (now  the  Government  Accountability 
Office)  stated  that  there  were  few  metrics  used  within  the  DoD  to  gauge  the  impact  of 
investments  or  the  effectiveness  of  processes  used  to  develop  and  transition 
technologies,  and  additional  metrics  in  technology  transition  were  needed  (GAO,  1999). 
In  2002,  the  GAO  further  articulated  that  the  DoD  needed  to  enable  success  through  the 
demonstration  of  value  and  the  credibility  of  new  processes  using  metrics  (GAO,  2002). 
To  address  these  compounding  challenges,  in  1999,  the  DoD  began  implementing  the 
Technology  Readiness  Level  (TRL)  as  a  metric  to  assess  the  maturity  of  a  program’s 
technologies  before  its  system  development  began.  Additionally,  the  DoD  made 
constructive  changes  to  its  approaches  to  acquisition  that  would  address  these  issues  by 
ensuring  a  weapon  systems’  technologies  are  demonstrated  to  a  high  level  of  maturity 
before  beginning  its  program  and  using  an  evolutionary  or  phased  approach  to 
developing  such  systems  (GAO,  2006).  More  recently,  the  successful  implementation  of 
TRL  within  the  DoD  has  been  noted  by  the  GAO  (Sullivan,  2010). 

Even  with  the  implementation  of  new  processes  and  practices  within  DoD  acquisition 
and  systems  engineering,  the  challenges  are  still  significant.  Nowhere  is  the  need  for 
enhanced  monitoring  capabilities  more  visible  for  systems  than  in  the  development  of 
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complex  systems.  Thus  far,  the  TRL  scale  has  been  a  key  gauge  of  the  current  status  of 
maturity  of  a  given  technology  and  even  systems  within  DoD  by  monitoring  capability 
development  from  concept  definition  through  operations  and  support.  In  countless 
development  efforts  TRL  has  been  a  key  indicator  of  progress  and  aided  dramatically  in 
keeping  programs  on  track  and  adjudicating  the  perceived  maturity  of  a  technology  for 
acquisition  into  a  program.  It  has  additionally  been  incorporated  as  a  decision  criterion 
in  the  Defense  Acquisition  Milestone  process  (see  DoD  5000.02)  along  with  other 
government  agency  guidance,  i.e.  National  Aeronautics  and  Space  Administration 
Systems  Engineering  Handbook  (NASA/SP-2007-6105). 

Consequently,  despite  the  utility  and  value  of  the  TRL  as  a  metric  for  determining 
technology  maturity  before  transitioning  into  a  system,  TRL  was  not  intended  to 
address  system  integration  or  to  indicate  that  the  technology  will  result  in  successful 
development  of  a  system.  Additionally,  when  TRL  is  applied  to  components  within  a 
complex  system,  the  model  of  using  individual  technology  maturity  as  a  measure  of 
readiness  to  integrate  into  system  development  can  become  confounded.  Similar 
problems  also  become  apparent  with  many  other  technology  development  tools  when 
applied  in  a  systems  context.  These  challenges  and  limitation  of  TRL  were  expressed  by 
the  Honorable  Ashton  Carter,  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  (AT&L),  in  his  Memorandum  for  Acquisition  Professionals 
entitled,  “Better  Buying  Power:  Guidance  for  Obtaining  Greater  Efficiency  and 
Productivity  in  Defense  Spending.”  He  states, 

Reform  TRL  reviews  to  focus  on  technology  as  opposed  to  engineering 
and  integration  risk.  The  TRL  review  and  certification  process  has  gown 
well  beyond  the  original  intent  and  should  be  reoriented  to  an  assessment 
of  technology  maturity  and  risk  as  opposed  to  engineering  or  integration 
risk.  I  am  directing  the  DDR&E  to  review  this  process  and  to  make 
recommendations  to  refocus  the  TRL  certification  process  to  be  consistent 
with  its  original  intent. 

While  the  TRL  has  been  well  proven  for  its  effectiveness  in  gauging  individual 
technology  maturity  in  research  and  development  applications,  its  extrapolation  to  more 
complex  systems  integration  (e.g.,  SoS),  dictated  by  emerging  DoD  requirements,  brings 
about  a  host  of  issues.  By  looking  only  at  the  status  of  individual  component  technical 
maturity,  TRL  fails  to  account  for  the  complexities  involved  in  the  integration  of  these 
components  into  a  functional  system  and  creates  the  opportunity  for  performance  gaps 
to  remain  hidden  until  late  in  the  development  cycle.  In  other  words,  application  of  TRL 
to  systems  of  technologies  is  not  sufficient  to  give  a  holistic  picture  of  system  readiness 
since  TRL  is  only  a  measure  of  an  individual  technology.  Furthermore,  assessments 
taking  into  account  several  technologies  rapidly  become  very  complex  without  a 
systematic  method  of  comparison  of  the  technologies  and  their  status  as  they  relate  to 
one  another.  Finally,  multiple  TRLs  do  not  provide  insight  into  integration  between 
technologies  or  the  maturity  of  the  resulting  system.  This  monitoring  of  integration 
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status  is  critical  as  it  has  been  repeatedly  shown  that  most  complex  system  development 
efforts  fail  at  the  integration  points.  This  lack  of  insight  and  the  need  to  provide  a 
method  for  monitoring  of  integration  status  led  to  the  development  of  complementary 
concepts  (e.g.  Integration  Readiness  Level,  Manufacturing  Readiness  Level)  that 
expounds  on  the  traditional  TRL  with  the  development  of  other  criteria  to  gain  a  more 
complete  perspective  of  system  maturity. 

Previous  research  conducted  by  the  SysDML  has  begun  the  development  of  a  Systems 
Maturity  Assessment  (SMA)  methodology  that  pairs  the  traditional  TRL  scale  with  an 
additional  series  of  criteria  known  as  the  Integration  Readiness  Level  (IRL)  to  gain  a 
more  complete  perspective  of  system  maturity.  The  foundation  of  the  SMA  methodology 
considers  each  technology,  but  instead  of  being  a  stand-alone  metric  for  determining 
readiness,  it  is  analyzed  in  concert  with  both  its  integration  requirements  and  the 
maturity  of  other  technologies  with  which  it  interfaces.  In  addition,  this  approach  is 
intended  to  gain  insight  into  potential  risks  related  to  developmental  maturity,  but  not 
to  be  a  tool  for  making  engineering  design  risk  decisions. 

A  process  has  been  developed  that  uses  the  SMA  methodology  with  a  measurement  (i.e. 
System  Readiness  Level)  to  assess  the  maturity  of  a  system  under  development  and 
make  program  decisions  to  capitalize  on  this  information.  The  SMA  methods  have  been 
implemented  most  notably  by  the  US  Navy  PMS  420,  Littoral  Combat  Ship  Mission 
Module  Program  Office.  SMA  has  been  highly  successful  on  the  program  and  has  paid 
dividends  in  terms  of  both  increasing  decision  maker  visibility  into  true  system  status 
and  allowing  for  pre-emptive  actions  to  be  taken  to  mitigate  potential  developmental 
issues.  The  program  office  is  now  looking  at  building  upon  the  current  SMA 
methodology  and  expanding  it  to  new  uses  in  guiding  technology  selection,  insertion 
and  tradeoffs,  as  well  as  for  use  in  cost  modeling  to  understand  the  impacts  of 
implementing  technology  options.  This  research  and  report  describe  some  of  these 
efforts. 


2.2  Existing  Acquisition  Environment  and  Guidance 

In  program  management  for  acquisition,  resources  are  frequently  allocated  with  the 
purpose  of  executing  tasks  to  maintain  schedule  and  budget.  This  can  distract  from  the 
ultimate  objective  of  developing  a  product  (or  system)  to  satisfy  a  customer.  A 
fundamental  challenge  to  solving  this  problem  is  that  when  attempting  to  meet  the 
emergent  needs  of  the  warfighter,  program  managers  will  often  continue  the 
development  of  a  system  through  the  acquisition  lifecycle— even  while  they  coordinate 
the  design  activities  with  preliminary,  ambiguous,  or  subjective  information.  The 
balance  between  customer  needs  (e.g.,  warfighter)  and  design  activities  for  acquisition 
can  create  a  delicate  balance  between  the  overview  required  by  the  program  manager  to 
make  strategic  acquisition  decisions  and  the  detail  that  is  the  focus  of  the  system 
developers  and  engineers. 
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In  2009,  the  One  Hundred  Eleventh  Congress  of  the  US  enacted  legislation  that  was 
focused  on  the  improvement  of  the  organization  and  procedures  of  the  DoD  for  the 
acquisition  of  major  weapon  systems.  This  act  specifies  the  establishment  of  positions 
and  activities  within  the  DoD  for  assessment  of  technology  maturity,  consideration  of 
trade-offs  among  cost,  schedule,  and  performance  objectives,  and  identification  and 
mitigation  of  systemic  problems  in  major  defense  acquisition  programs  prior  to 
Milestone  B  approval.  In  an  ongoing  effort  for  acquisition  reform,  the  DoD  has  also 
released  the  Technology  Readiness  Assessment  (TRA)  Deskbook  and  issued  and 
updated  DoD  Instruction  5000.02  and  DoD  Directive  5000.01. 

For  example,  DoD  Instruction  5000.02  describes  the  System  Capability  and 
Manufacturing  Process  Demonstration,  which  is  intended  to  demonstrate  the  ability  of 
the  system  to  operate  in  a  useful  way  consistent  with  the  approved  key  performance 
parameters  (KPPs)  and  that  system  production  can  be  supported  by  demonstrated 
manufacturing  processes.  Developmental  test  and  evaluation,  early  operational 
assessments,  and,  where  proven  capabilities  exist,  the  use  of  modeling  and  simulation  to 
demonstrate  system/system-of-systems  integration  is  critical  during  this  effort. 

In  addition,  5000.02  requests  the  program  manager  to  prepare  an  Acquisition  Strategy 
to  guide  activity  during  Engineering  and  Manufacturing  Development  (EMD).  Among 
other  requirements,  the  Acquisition  Strategy  must  describe  the  relationship  and 
associated  dependencies  with  other  system  elements  if  a  program  is  part  of  a  system-of- 
systems  or  family-of-systems. 

Additional  DoD  References  and  Guidance  on  Acquisition: 

•  DoD  Directive  5000.01  The  Defense  Acquisition  System  -  Provides  management 
principles  and  mandatory  policies  and  procedures  for  managing  all  acquisition 
programs 

•  DoD  Instruction  5000.02  Operation  of  the  Defense  Acquisition  System  - 
Establishes  a  simplified  and  flexible  management  framework  for  translating 
capability  needs  and  technology  opportunities,  based  on  approved  capability 
needs,  into  stable,  affordable,  and  well-managed  acquisition  programs  that 
include  weapon  systems,  services,  and  automated  information  systems. 

•  Defense  Acquisition  Guidebook  Chapter  4.3.2.4.24  Technology  Readiness 
Assessment  -  A  systematic,  metrics-based  process  that  assesses  the  maturity  of 
critical  technology  elements  (CTEs),  including  sustainment  drivers. 

•  Introduction  to  Defense  Acquisition  Management,  Defense  Acquisition 
University 

•  Technology  Readiness  Assessment  Deskbook  DUSD(S&T)  -  Description  of 
suggested  best  practices,  roles,  and  procedures  for  meeting  the  Technology 
Readiness  Assessment  requirements  of  the  Defense  Acquisition  System. 
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2.3  Maturity  Monitoring  Practices 

There  have  been  many  attempts  to  identify  alternative  readiness/maturity  levels  that 
will  complement  TRL,  such  as  Design  Readiness  Level,  Manufacturing  Readiness  Level, 
Software  Readiness  Level,  and  Operational  Readiness  Level.  While  the  DoD  has  relied 
on  some  of  these  subjective  assessment  techniques  for  developing  the  program 
overview,  which  then  becomes  the  basis  for  making  strategic  acquisition  decisions,  these 
subjective  assessments  are  labor-intensive,  error-prone,  and  inadequate  for  the  desired 
management  controls.  Notwithstanding  the  limitations  of  many  of  these  metrics,  any 
metric  should  not  lose  sight  of  what  makes  it  effective  and  efficient  in  an  organization 
(Dowling  and  Pardoe,  2005): 

1.  The  way  the  value  is  used  should  be  clear. 

2.  The  data  to  be  collected  for  the  metric  should  be  easily  understood  and  easy  to 
collect. 

3.  The  method  of  deriving  the  value  from  the  data  should  be  clear  and  as  simple  as 
possible. 

4.  Those  for  whom  the  use  of  the  metric  implies  additional  cost  should  see  as  much 
direct  benefit  as  possible  (i.e.,  collecting  the  data  should  not  cost  more  than  its 
value  to  the  decision  process). 

Fundamentally,  these  controls  should  be  based  on  system  and  acquisition  attributes  that 
can  be  quantitatively  measured  using  system  metrics.  The  tension  between  subjectivity 
and  detail  is  rationalized  through  prescriptive  techniques,  which  allow  people  to  make 
better  decisions  by  using  normative  models,  but  with  knowledge  of  the  limitations  and 
descriptive  realities  of  human  judgment.  Thus,  further  detail  is  needed  into  the 
assessment  and  use  of  some  of  these  metrics  in  order  to  limit  the  subjectivity  and 
increase  the  rigor  in  their  use.  It  is  also  important  to  note  that  a  variety  of  tools  and 
approaches  should  be  applied  to  gauge  overall  development  status.  The  TRA  Deskbook 
notes  that,  “the  TRA  should  not  be  the  sole  means  of  discovering  technology  risk,” 
meaning  other  scales  must  be  applied  in  order  to  adequately  capture  system  risk. 
Therefore,  while  this  research  and  supporting  report  address  some  of  these  challenges 
for  assessing  maturity  of  system  development,  there  is  still  significant  work  to  be  done 
to  increase  the  reliability,  efficiency,  and  effectiveness  of  system  maturity  assessment. 


3  SRL  FOUNDATIONS 


Given  the  emerging  requirements  for  a  measure  of  complex  system  readiness,  in  2006 
the  SysDML  at  Stevens  Institute  of  Technology  presented  the  concept  of  a  System 
Readiness  Level  for  managing  system  development  (Sauser  et  al.,  2006).  As  a  result,  in 
2007  the  SysDML  in  collaboration  with  the  US  Navy  PMS  420/SPAWAR  and  Northrop 
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Grumman  Corporation  were  chartered  to  define  a  system  maturity  scale  and  supporting 
methodology.  The  core  requirements  included  that  the  scale  must  be  robust,  repeatable, 
and  agile  so  outputs  could  not  only  be  trusted  and  replicated,  but  that  the  methodology 
as  a  whole  be  easily  transferred  to  a  variety  of  different  applications  and  architectures. 
In  response  to  this  challenge,  the  concept  of  a  System  Readiness  Level  (SRL)  that  would 
incorporate  a  TRL  and  an  Integration  Readiness  Level  (IRL)  was  developed  as  depicted 
in  Figure  l  (Sauser  et  al.,  2006) 


The  System 


t 

SRL  =  /  (TRL,  IRL) 

Figure  l:  System  Readiness  Level 


Similar  to  TRL,  the  IRL  was  defined  as  a  series  of  levels  that  articulate  the  key 
maturation  milestones  for  integration  activities.  The  introduction  of  an  IRL  to  the 
assessment  process  not  only  provided  a  check  as  to  where  a  technology  was  on  an 
integration  readiness  scale  but  also  presented  a  direction  for  improving  integration  with 
other  technologies.  Just  as  a  TRL  is  used  to  assess  the  risk  associated  with  developing 
technologies,  the  IRL  is  designed  to  assess  the  risk  associated  with  integrating  these 
technologies.  Building  upon  similar  efforts  to  define  an  integration  maturity  scale,  the 
IRL  has  been  refined  to  include  nine  levels  as  represented  in  Table  1.  For  more  details 
on  the  formulation  of  the  IRL  see  (Sauser  et  al.,  2010).  Appendix  A  contains  a  further 
breakdown  of  the  activities  associated  with  each  IRL  level,  and  a  Subject  Matter 
Assessment  of  the  criticality  of  these  activities  as  they  relate  to  their  respective  level. 
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Table  l:  Integration  Readiness  Level 


IRL 

Definition 

Description 

Pragmatic 

9 

Integration  is  Mission  Proven  through 
successful  mission  operations. 

IRL  9  represents  the  integrated  technologies  being  used  in  the  system 
environment  successfully.  In  order  for  a  technology  to  move  to  TRL  9 
it  must  first  be  integrated  into  the  system,  and  then  proven  in  the 
relevant  environment,  so  attempting  to  move  to  IRL  9  also  implies 
maturing  the  component  technology  to  TRL  9. 

8 

Actual  integration  completed  and  Mission 
Qualified  through  test  and  demonstration, 
in  the  system  environment. 

IRL  8  represents  not  only  the  integration  meeting  requirements,  but 
also  a  system-level  demonstration  in  the  relevant  environment.  This 
will  reveal  any  unknown  bugs/defect  that  could  not  be  discovered  until 
the  interaction  of  the  two  integrating  technologies  was  observed  in  the 
system  environment. 

Syntactic 

7 

The  integration  of  technologies  has  been 
Verified  and  Validated  and  an 
acquisition/insertion  decision  can  be 
made. 

IRL  7  represents  a  significant  step  beyond  IRL  6;  the  integration  has  to 
work  from  a  technical  perspective,  but  also  from  a  requirements 
perspective.  IRL  7  represents  the  integration  meeting  requirements 
such  as  performance,  throughput,  and  reliability. 

6 

The  integrating  technologies  can  Accept, 
Translate,  and  Structure  Information 

for  its  intended  application. 

IRL  6  is  the  highest  technical  level  to  be  achieved,  it  includes  the 
ability  to  not  only  control  integration,  but  specify  what  information  to 
exchange,  unit  labels  to  specify  what  the  information  is,  and  the  ability 
to  translate  from  a  foreign  data  structure  to  a  local  one. 

5 

There  is  sufficient  Control  between 
technologies  necessary  to  establish, 
manage,  and  terminate  the  integration. 

IRL  5  simply  denotes  the  ability  of  one  or  more  of  the  integrating 
technologies  to  control  the  integration  itself;  this  includes  establishing, 
maintaining,  and  terminating. 

4 

There  is  sufficient  detail  in  the  Quality 
and  Assurance  of  the  integration 
between  technologies. 

Many  technology  integration  failures  never  progress  past  IRL  3,  due  to 
the  assumption  that  if  two  technologies  can  exchange  information 
successfully,  then  they  are  fully  integrated.  IRL  4  goes  beyond  simple 
data  exchange  and  requires  that  the  data  sent  is  the  data  received 
and  there  exists  a  mechanism  for  checking  it. 

Semantic 

3 

There  is  Compatibility  (i.e.  common 
language)  between  technologies  to 
orderly  and  efficiently  integrate  and 
interact. 

IRL  3  represents  the  minimum  required  level  to  provide  successful 
integration.  This  means  that  the  two  technologies  are  able  to  not  only 
influence  each  other,  but  also  communicate  interpretable  data.  IRL  3 
represents  the  first  tangible  step  in  the  maturity  process. 

2 

There  is  some  level  of  specificity  to 
characterize  the  Interaction  (i.e.  ability  to 
influence)  between  technologies  through 
their  interface. 

Once  a  medium  has  been  defined,  a  “signaling”  method  must  be 
selected  such  that  two  integrating  technologies  are  able  to  influence 
each  other  over  that  medium.  Since  IRL  2  represents  the  ability  of  two 
technologies  to  influence  each  other  over  a  given  medium,  this 
represents  integration  proof-of-concept. 

1 

An  Interface  between  technologies  has 
been  identified  with  sufficient  detail  to 
allow  characterization  of  the  relationship. 

This  is  the  lowest  level  of  integration  readiness  and  describes  the 
selection  of  a  medium  for  integration. 

IRL  is  a  systematic  measurement  of  the  interfacing  of  compatible  interactions  for 
various  technologies  and  the  consistent  comparison  of  the  maturity  between  integration 
points.  For  further  clarification,  the  nine  levels  of  IRL  presented  in  Table  l  can  be 
understood  as  having  three  stages  of  integration  definition:  semantic,  syntactic,  and 
pragmatic.  Semantics  is  about  relating  meaning  with  respects  to  clarity  and 
differentiation.  Thus  IRL  1-3  are  considered  fundamental  to  describing  what  we  define 
as  the  three  principles  of  integration:  interface,  interaction,  and  compatibility.  We 
contend  that  these  three  principles  are  what  define  the  subsistence  of  an  integration 
effort.  The  next  stage  is  Syntactic,  which  is  defined  as  a  conformance  to  rules.  Thus 
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IRLs  4-7  are  about  assurance  that  an  integration  effort  is  in  compliance  with 
specifications.  The  final  stage  is  Pragmatic,  which  relates  to  practical  considerations. 
Thus,  IRLs  8-9  are  about  the  assertion  of  the  application  of  an  integration  effort. 

With  the  ability  to  assess  both  the  technologies  and  integration  elements  along  a 
numerical  maturation  scale,  the  next  challenge  was  to  develop  a  metric  that  could  assess 
the  maturity  of  the  entire  system  under  development.  Therefore,  the  SRL  has  been 
described  using  a  normalized  matrix  of  pair-wise  comparisons  of  TRLs  and  IRLs  for  any 
system  under  development  could  yield  a  measure  of  system  maturity  (Sauser  et  al., 
2008b).  The  rationale  behind  the  SRL  is  that  in  the  development  lifecycle,  one  would  be 
interested  in  addressing  the  following  considerations: 

•  Quantifying  how  a  specific  technology  is  being  integrated  with  every  other 
technology  to  develop  the  system. 

•  Providing  a  system-wide  measurement  of  readiness. 

Under  this  method,  TRL  evaluations  for  each  technology  and  IRL  evaluations  of  each 
integration  are  combined  using  matrix  mathematics  (explained  in  detail  later)  to 
produce  a  comprehensive  assessment  where  each  technology  within  the  system  is 
weighted  according  to  all  of  its  integrations  and  then  rolled  up  to  a  system  level.  It  is 
important  to  emphasize  that  the  SRL  is  not  a  quantitatively  defined  rating  system,  but  is 
instead  an  analytical  combination  of  the  TRL  and  IRL  scales.  In  others  words,  the  SRL 
output  is  purely  a  function  of  the  TRL  and  IRL  inputs. 

Imperative  to  the  development  of  the  SRL  have  been  the  supporting  methodologies  for 
using  the  SRL  in  the  practice  of  systems  development  and  acquisition.  Thus,  SysDML  in 
conjunction  with  the  US  Navy  PMS  420/SPAWAR  and  Northrop  Grumman  have 
incorporated  SRL  into  an  approach  for  the  assessment  of  system  maturity,  as  described 
in  this  report.  While  readiness  is  defined  as  a  scale,  maturity  is  defined  as  the  practices 
that  support  the  development  (or  maturation)  of  a  system’s  readiness. 

Therefore,  SRL  is  more  than  purely  a  qualitative  assessment.  It  requires  the  user  to 
define  the  element  level  contributions  of  the  multiple  technologies  and  integrations  that 
make  up  the  system.  In  this  way,  it  allows  managers  to  evaluate  system  development  in 
real-time  and  take  proactive  measures  by  examining  the  status  of  all  elements  of  the 
system  simultaneously.  Furthermore,  the  methodology  is  adaptive  to  use  on  an  array  of 
system  engineering  development  efforts  and  can  also  be  applied  as  a  predictive  tool  for 
technology  insertion  trade  studies  and  analysis. 


3.1  SRL  Calculation 

Under  this  method,  the  evaluation  of  technology  using  TRL  and  the  evaluation  of  each 
integration  using  IRL  are  combined  via  a  set  of  mathematical  formulas  (explained  in 
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detail  later)  to  produce  an  integrated  assessment  where  each  technology  within  the 
system  is  weighted  according  to  all  of  its  integrations  and  then  calculated  at  a  system 
level.  It  is  important  to  emphasize  that  the  SRL  is  not  a  quantitatively  defined  rating 
system,  but  is  instead  an  analytical  combination  of  the  TRL  and  IRL  scales.  A 
fundamental  assertion  in  the  calculation  of  the  SRL  is  our  interpretation  and  use  of 
these  inputs.  We  assert  that  the  TRL  and  IRL  inputs  are  purely  data  inputs  into  the  SRL 
calculation,  and  we  do  not  assert  any  specific  form  of  numerical  scale,  i.e.  nominal, 
ordinal,  interval,  or  ratio.  Thus,  to  assert  a  form  of  scale-conversion  on  the  inputs 
prepositions  that  the  origin  of  the  data  type  is  known  or  natural.  Lord  (1953)  describes 
this  as,  “the  numbers  do  not  know  where  they  came  from.”  Likewise,  Gaito  (1980) 
states  that  there  are  some  fundamental  misconception  in  the  conversion  or  use  of  data 
for  mathematical  purposes.  He  explains  that  the  four  scales  or  measurement  levels 
introduced  by  Stevens  (1946)  are  a  “confusion  of  measurement  theory  and  statistical 
theory”  (Gaito,  1980).  Since  TRL  and  IRL  are  scales  of  non-natural  origin,  their 
interpretations  of  forms  of  scale  are  also  interpretable  by  the  user  of  the  data.  Thus,  we 
make  9=9,  8=8,  7=7,  etc.  This  conversion  is  justifiable  if  the  conversion  does  not  alter 
the  scale  (e.g.  8  does  not  become  more  important  than  9  or  7  less  important  than  6) 
(Shah  and  Madden,  2004,  Akritas,  1990).  As  discussed  in  Sauser,  et  al.  (Sauser  et  al., 
2008b)  and  Magnaye,  et  al.  (Magnaye  et  al.,  2010),  the  use  of  scale  data  in 
mathematically  assessing  progress  or  status  without  scale-conversion  is  not  without 
precedence,  i.e.  Grade  Point  Average  (GPA),  Analytical  Hierarchy  Process  (AHP)  and 
Failure  Mode  Effects  and  Criticality  Analysis  (FMECA).  In  continued  research  by  the 
SysDML,  the  degree  of  variation  is  been  investigated  that  will  allow  for  the  TRL,  IRL, 
and  SRL  values  to  move  from  ordinal  to  nominal  data. 


The  computation  of  the  SRL  is  a  function  of  the  TRL  and  IRL  matrices: 

•  Matrix  TRL  provides  a  blueprint  of  the  state  of  the  system  with  respect  to  the 
readiness  of  its  technologies.  TRL,  defined  as  a  vector  with  n  entries,  is  defined 
in  Equation  1,  where  TRLi  is  the  TRL  of  technology  i. 


[TRL  ]„xl 


TRLX 

TRL2 


TRL 


n 


•  Matrix  IRL  illustrates  how  the  different  technologies  are  integrated  with  each 
other  from  a  system  perspective.  For  a  system  with  n  technologies,  [IRL]  is 
defined  in  Equation  2,  where  IRLij  is  the  IRL  between  technologies  i  andj.  The 
hypothetical  integration  of  a  technology  i  to  itself  is  denoted  by  IRLii. 
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IRLn 

IRLn  • 

-  IRLu 

IRL2\ 

IRL  22 

..  IRLln 

IRLnl 

IRLn2  . 

••  iRLnn 

Briefly  stated,  the  IRL  matrix  is  obtained  as  a  symmetric  square  matrix  (of  size  nxn)  of 
all  possible  integrations  between  any  two  technologies  in  the  system.  For  technology 
integration  to  itself,  perfect  integration  is  assumed  (IRL=  9)  while  an  IRL  of  zero  is  used 
when  there  is  no  integration  between  two  elements.  On  the  other  hand,  the  vector  TRL 
defines  the  readiness  level  of  each  of  the  technologies  in  the  system.  The  calculation  of 
the  SRL  has  also  gone  through  a  series  of  refinements  and  the  most  recent  thorough 
discussion  has  been  presented  by  Sauser  et  al.  (2008a).  As  a  result  of  this  research 
another  minor  modification  to  the  SRL  calculation  is  the  re-naming  the  SRL  vector  (i.e. 
SRL,)  to  ITRL,.  ITRL,  indicates  the  maturity  of  technology  i  with  its  integrations 
considered.  With  a  system  comprised  of  n  technologies,  it  is  mathematically  described 
as 


(IRLX  {TRLX  +  IRLl2TRL2  + . . .  +  IRLlnTRLn )  /  m, 

{IRL,  ]77?L]  +  IRL,,TRL,  + ...  +  IRL,  nTRL  n )  /  in, 

(IRLn\TRL\  +  IRLn2TRL2  + ...  +  IRLnnTRLn  )/mn  where  IRLij=IRLji, 

Where  m,is  the  number  of  integrations  with  technology  i  plus  its  integration  to  itself. 
With  the  ability  to  assess  both  the  technologies  and  integration  elements  along  a 
numerical  maturation  scale,  the  next  challenge  was  to  develop  a  metric  that  could  assess 
the  maturity  of  the  entire  system  under  development.  Therefore,  the  SysDML  has 
described  how  using  a  normalized  matrix  of  pair-wise  comparisons  of  TRLs  and  IRLs 
for  any  system  under  development  could  yield  a  measure  of  system  maturity.  SRL  is 
then  calculated  as 

SRL  ITRLi+ITRL2  +-  +  ITRLn 
n 

We  will  explain  in  section  4.1.2  SRL  MAPPING  TO  DEVELOPMENT 
ACTIVITIES/STATUS  how  the  data  from  the  SRL  method  is  best  interpreted. 


[ITRL]  = 


ITRLX 

ITRL, 


ITRL, 


3.2  SRL  Alternatives 

While  we  have  investigated  multiple  was  of  formulating  an  SRL  from  TRL  and  IRL 
inputs,  we  never  contend  that  the  SRL  calculation  just  presented  is  the  only  way  to 
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effectively  utilize  TRL  and  IRL  data.  We  have  found  that  this  method  is  sufficient  and 
has  shown  great  applicability  and  insight  to  developmental  risks  in  systems  engineering. 
That  said,  there  are  alternative  approaches  to  using  TRL  and  IRL  data  to  formulate  a 
SRL.  With  that,  as  part  of  this  research  we  describe  a  different  approach  to  determining 
the  SRL  that  extends  the  method  proposed  by  Ramirez-Marquez  and  Sauser  (2009)  in 
their  paper  "System  Development  Planning  via  System  Maturity  Optimization."  As 
stated,  the  SRL  index  incorporates  both  the  TRL  and  IRL  in  order  to  provide  a  measure 
of  the  maturity  of  a  system.  All  these  measures,  IRL,  TRL  and  SRL,  take  integer  values 
between  [0,9],  in  their  standard  versions,  or  real  values  between  [0,1]  in  their 
normalized  versions.  The  idea  behind  SRL  is  to  use  it  as  a  decision  support  approach, 
via  resource  allocation,  in  order  to  assess  the  system's  developmental  maturity  and 
lifecycle  position. 

The  goal  of  this  sub-task  was  to  look  at  an  alternative  way  of  determining  the  SRL  index 
using  tropical  geometry,  in  particular  min-plus  algebra.  This  algebra  is  an  attractive  tool 
for  computing  SRL  when  a  fundamental  premise  is  that  a  system  cannot  be  "more 
ready"  than  the  "less  ready"  of  its  sub-systems;  this  is  what  min-plus  algebra  will  reflect 
when  applied  to  this  particular  situation.  Throughout  all  the  calculations  in  the 
presentation  of  the  min-plus  algebra  we  use  the  normalized  versions  of  the  SRL,  TRL 
and  IRL,  for  simplicity  purposes. 

We  show  two  particular  cases  when  there  are  n  technologies  with  IRL  equal  to  zero  (i.e. 
the  technologies  are  fully  disconnected)  and  when  the  n  technologies  are  fully  connected 
(i.e.  IRL=9  or  1  in  its  normalize  version).  We  provide  a  short  introduction  to  min-plus 
algebra  as  well  as  its  application  to  the  cases  described  above.  Lastly  we  present  an 
example  using  only  two  technologies  for  the  cases  when  IRL=o,  IRL=9  and  when  the 
two  technologies  have  the  same  fixed  level  of  technology. 


3.2.1  Min-Plus  Algebra 

In  order  to  provide  a  better  understanding  of  our  computations  in  further  sections  we 
start  with  a  short  overview  of  the  min-plus  algebra.  For  a  more  comprehensive  review  of 
the  min-plus  algebra  the  reader  is  referred  to  Shutter  and  Moor  (1997)  and  Cohen  et  al. 
(1999);  some  applications  can  be  found  in  Heidesgott  et  al.  (2006)  and  Mikhalkin 
(2006). 

A  min-plus  algebra  is  a  semi-ring  with  the  algebraic  structure  P  min  =  (s.Kmm,©',®,e, A) 

5 

where, 

•  <2  =  0 

•  e'  =  00 

•  ...  =  -R {e'\  gt  are  the  real  numbers. 

•  ©' :  s*Hmm  x ->  91m,„ ,  such  that,  given  a,b  <=  Tirn  ,  n 
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a(£>b=m  {ca,b>$i 

•  ® :  9*min  x  sJimin  ->  <Rmin ,  such  that,  given  a,  b  e 

a(E>b=ci+b 


The  operation  ©'  and  ®  have  several  algebraic  properties  such  as  associativity, 
commutativity,  distributivity  of  ®  over  ©' ;  among  others. 

For  matrices  A  e  LR^^nd  Vlthen  matrix  product  xf(x)A?is  defined  by 

r®«,  * 

7=1 

=  m  i{ff,  } 

j^P 

for  i  e  n  and  k  e  m  .  Note  that  This  is  just  like  in  the  regular  linear  algebra 

with  "  +  "  replaced  by "  min"  and  "  x "  by "  + 


3.2.2  n  Technologies  with  IRL=0  in  the  Min-Plus  Sense 

In  this  section  we  present  a  different  procedure  for  computing  the  SRL.  Although  the 
procedure  looks  almost  the  same  than  the  one  presented  in  section  3.2.1,  the 
computations  and  the  results  will  be  different  due  to  the  use  of  the  min-plus  algebra.  Let 

us  start  by  computing  the  vector  y=[I  fppr  R\  Awhere  \TLR]  =  \k}  k2  ■■■  kn f . 
y=[I 

=  Ink 

min{l  +  kx,k2,...,kn} 

_  min{&1,l  +  &2, ...,£„} 

mm{kl,k2,...,l  +  kn}  _ 

We  now  define  the  SRL  as  the  /2  -norm  of  y  in  the  min-plus  sense,  i.e. 

SRL=  \y\2=  (J)y, 

i= 1 

Therefore,  for  n  Technologies  with  IRL=  Q  the  SRL  is  given  by 

SRL  =  mm^{2\rm{\  +  kx,k2,...,kJ,...,2rm\{kx,k2,...,\+kn) 

=  2vcm{\rm(\  +  kx,k2,...,kn),...,vcm(kx,k2,...,\+kn)} 

=2min(k]  k2,  kn) 
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3.2.3  n  Technologies  with  IRL= 9  in  the  Min-Plus  Sense 

In  the  case  where  we  have  n  technologies  with  IRL=  9,  we  found,  in  section  3.2.2,  that 
the  [IRL]  was  given  by  a  matrix  having  all  its  elements  equal  to  1.  Therefore,  by  setting 

\TLR\  =  [kx  k2  -  kj  ,  we  get  that  the  vector  y  is  given  by  the  following  expression. 
y=[I  7Jx£T  ig  L 


f\  1 

1 

n 

kx 

1  1 

1 

1 

k-, 

vl  1 

1 

b 

K_ 

min{l  +  kx,  1  +k2,...,i-kn} 
min{l  +kx,  1  +  k2,...,i-kn} 

min{l  +kx,  1  +k2,...,\-kl^) 

Thus, 

SRL  =|  y  \2 =  ©  yt  ®  yt  =  2  +  2  min  {kx ,  k2 kn } 

1=1 

Note  that  equations  (11)  and  (15)  suggest  that  in  this  case  (i.e.  in  the  min-plus 
sense),  having  n  technologies  with  IRL=0  or  IRL=9  will  make  a  difference,  in 
comparison  to  the  case  presented  in  sections  3.2.1  and  3.2.3. 


3.2.4  Min-Plus  method  in  general 


Therefore,  by  setting  [TLR\  =  \kx  k2 
by  the  following  expression. 

y=v 


ID 


Thus, 


'  1 

al,2 

«1  ,n 

k\ 

a2\ 

1  ... 

a2,n 

k2 

M 

M  O 

M 

M 

1 

an,2  - 

an,ny 

kn_ 

min{  1  +kx,ax2 

+  k2,.. 

•Ai,„ 

vam{a2X+kx,\  +  k2,...,aVl  +kn} 

M 

min{a„  4  +kx,an2+k2,...,l+kn} 


SRL  =|  y 

□ 


0 Vi  ®yt  =  2m i  ijim  i  friL/  +k  - } } 

<>i;  ./=i; 
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Therefore,  the  Min-Plus  method  gives  the  result  of  the  minimum  of  the  sum  of  TRLi  and 
the  lowest  IRL  that  between  Technology  i  and  any  other  technologies 


Note  that  kf  <al  f  +k/  <1  +kf 

|  AybchnobgysNOTconnecfcdbATLEASTONEobhet  chnobges 
[l  +  AyEchnobgys  JVDSTMVTURELYconnecbdwiiEVERYohebchnobgy 


aij  +  kj  = 


So  we  have 


□ 


SRL=2rm  {nil  \alJ  +A; }  }= 


L=L  ./=!; 


J  2nh(A[„  A^cor  responds  Os  cenar  b32^enf  u^ds  connecfed. 

|2+2nh(A;1„  A^cor  res  ponds  Os  cenar  b33^ienf  uyandncstmtir  ayconnecfed. 


3.2.5  Example:  Two  Technologies 

The  following  example  shows  how  the  model  works  in  the  case  where  the  system  has 
only  2  technologies.  We  disscuse  three  possible  scenarios: 


(fully  disconnected)  IRL=o  among  the  two  technologies. 

(fully  connected)IRL=9  among  the  two  technologies. 

The  same  fixed  technology  readiness  level  among  the  two  technologies,  i.e. 

tr^  nu. 


For  any  of  the  three  scenarios,  let  [IRL] 


' ax  a2' 

CL  ^  CL  | 

and  [TRL  ]  = 

1 

(N 

1 _ 

V  J 

,  then 


y=[I 


m  i{nj  +k x,a2  +k2} 
m  i{n2  +k^,a x  +A;2} 


l.  (fully  disconnected)  IRL=o  among  the  two  technologies. 

If  a2  =  0  ,  i.e.  /A£=Oamong  technologies,  then 

2 

I  v  I2  =  0  T,  ®  y,  =  min  {2 A, ,2 kx }  =  2  min  {k2 ,  A, } 

i= 1 


2.  (fully  connected) IRL =9  among  the  two  technologies. 

If  a2^ax ,  then 

2 

\y\2=  ®yt  =  min {2  +  2iriin{2A:1,2A:2}?2  +  2min{A:1,A:2}} 

1=1 

=  2+ 2  m 
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3.  The  same  fixed  technology  readiness  level  among  the  two 
technologies. 

If  kx=k2  =  k ,  then 

2 

|  y  |  =©7*  ®  yj  =  m i  r^2 m i  +  k, a2  +  ^),2mi^/1  +  k,a2  +  £)} 

i=i 

=  2  m  i  +  k,a2+  k)=  2  (a2  +  k)=  2  a2  +  2k 


□ 


\y\2=<§jyl<&yl  =m  y  ,  %-+k 
i= 1  I  Vh  J 


=  2  —  +  2k 

a. 


The  following  graph  shows  the  system  readiness  levels  for  the  different  technology 
readiness  levels  using  this  equation. 


Figure  2:  SRLs  for  the  different  TRLs  using  Min-Plus  SRL 


3.2.6  Comparison  of  SRL  to  Min-Plus  SRL 

In  this  section,  we  calculted  the  SRL  for  two  systems  that  consisted  of  the  same  fully 
maure  technologies  (all  TRL=9),  but  differ  in  the  following: 
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•  one  with  no  integration 

•  the  other  one  is  fully  connected,  and  all  IRL=9 

In  our  examples  we  identified  some  limitation  of  the  SRL  for  these  two  systems  because 
the  SRL  value  was  the  same,  and  then  presented  the  min-plus  method  that  can  solve  this 
problem.  However,  in  these  examples  we  assumed  that  we  had  no  knowledge  of  the 
system  architecture  but  was  only  looking  at  the  one-to-one  relationships.  The  SRL  is 
intended  to  be  used  when  the  architecture  of  a  system  is  already  known,  and  an  isolated 
technology  is  not  to  be  considered  part  of  the  system.  Under  this  assumption,  the  two 
presented  systems  are  different,  the  first  one  would  not  be  considered  a  system. 
Therefore,  our  argurment  in  in  this  case  is  invalid  when  considering  a  complete  system 
architecture.  It  only  tells  us  that  the  SRL  does  not  account  for  the  process  when  an 
architecture  is  from  unknown  to  known. 

From  the  general  formula  for  the  Min-Plus  SRL  method,  this  method  calculates  the  SRL 
as  the  minimum  of  the  sum  of  TRLi  and  the  lowest  IRL  that  between  Technology  i  and 
any  other  technologies.  The  Min-Plus  method  intends  to  use  the  lowest  component 
readiness  level  as  the  SRL,  through  which  to  find  the  component  whose  maturity  is 
lagging  the  most.  However  like  the  SysDML  SRL,  it  still  does  not  tell  which  TRL  is  the 
lowest  or  which  IRL  is  the  lowest.  Thus,  this  method  works  when  the  system  is  fully 
connected  (every  technology  is  connected  with  any  other  technologies),  and  identifies 
the  weakness  of  maturity  with  the  consideration  of  TRL  and  IRL.  However,  when  the 
system  is  not  fully  connected  which  is  more  often  in  reality,  we  do  not  know  if  the 
weakness  is  just  the  TRL  or  is  the  combination  of  TRL  and  IRL  because  any  non-zero 
IRL  will  be  eliminated  by  a  zero  IRL  in  the  min-plus  method.  To  solve  this  problem,  a 
simple  formula  can  be  used  to  define  SRL  as:  SRL=(min(TRLi),  min(IRLij)).  Table  2 
summarizes  the  comparison  of  the  two  methods. 
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Table  2:  Comparison  of  SRL  and  Min-Plus  SRL 


SDML  SRL 

Min-Plus 

Formula 

ITRL=IRL*TRL 

SRL=Average(ITRLi) 

y=  [IRL]*  [TRL] 

SRL  =1  y  l2  =  ff )yt  ®yt  =  2min{min{ai-  •  ■}} 

i,=l;  7=1;  J 

7=1 

Risk  Orientation 

Neutral 

Risk-Averse 

When  to  use 

The  architecture 
(technology/integration)  of  a 
system  is  known 

1.  Can  be  used  during  the  process  of  figuring  out 

system  architecture. 

2.  Can  be  used  for  a  known  architecture 

What  it  does 

Provide  results  that  take  into 
account  all  the  component  RLs 

Provide  result  of  the  minimum  of  the  sum  of  TRLi 
and  the  lowest  IRL  that  between  Technology  i  and 
any  other  technologies. 

Give  the  readiness  of 
a  technology  with  the 
consideration  of  all 
its  integrations? 

Yes 

No 

Identify  the  most 
immature 
component? 

No,  because  the  result  is 
averaged. 

Yes 

Manifest  the 
improvement  on  the 
other  aspects  expect 
the  lowest  RL? 

Yes,  the  improvement  of  other 
aspects  contributes  to  the  value 
of  ITRLs/SRL. 

No,  because  it  only  provides  the  lowest  value. 

Identify  the  lowest 
TRL? 

No 

No 

Identify  the  lowest 
IRL 

No 

No 

4  Unking  SRL  to  Architecture,  Development, 
Milestones,  and  Costs 


4.1  Aligning  SRL  Methods  with  Architectures 

In  understanding  what  the  SRL  method  means  to  systems  engineering  architecture,  we 
first  must  clarify  what  is  meant  by  maturity  and  readiness.  We  distinguish  these  two 
terms  as:  the  readiness  of  a  system,  technology,  or  integration  implies  how  ready  it  is  to 
be  deployed  on  a  numeric  scale;  and  maturity  is  the  characterization  of  the  physical 
development  that  is  quantified  by  the  readiness.  Thus,  readiness  is  a  scale,  and  maturity 
is  the  definition  of  each  level  in  the  scale.  We  also  distinguish  maturity  within  a  system 
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as  having  three  dimensions:  physical,  logical,  and  functional.  The  TRL,  IRL,  and  SRL 
are  the  maturity  of  the  physical  representation  of  a  system  or  its  physical  architecture. 
Thus,  the  first  step  in  applying  the  SRL  Method  is  to  understand  the  physical  system 
and  its  representative  architecture,  which  shows  the  system  design  broken  down  into  all 
its  constituent  elements  (i.e.,  subsystems  and  components).  Likewise,  this  architecture 
is  supported  by  its  functional  and  logical  representations.  With  respects  to  this 
research,  we  only  focus  on  the  physical  architecture.  As  described  in  the  Technology 
Readiness  Assessment  Deskbook:  “The  physical  architecture  includes  a  representation 
of  the  software  and  hardware  ‘products’  necessary  to  realize  the  concept.  The  physical 
architecture  forms  the  basis  for  design  definition  documentation  (e.g.,  specifications, 
baselines,  and  the  work  breakdown  structure  (WBS))”.  The  allocation  of  the  functional 
definition  of  the  system  to  the  physical  definition  of  the  system  completes  the  “design” 
of  the  system  and  the  definition  of  its  components. 

Conversely,  a  functional  definition  of  an  architecture  can  be  mapped  (or  allocated)  to  a 
physical  view  component.  Together  these  views  define  the  design  (model)  of  the  system. 
The  system  model  becomes  the  input  to  the  detailed  design  and  fabrication  of  the 
physical  components  that  comprise  the  system.  This  physical  view  establishes  all  of  the 
discrete  subsystems  and  components  that  are  procured  or  developed  to  produce  the 
system.  It  also  defines  the  interfaces  between  these  subsystems  and  components. 

The  physical  view  also  identifies  the  selection  of  technologies,  including  both  developed 
and  commercial,  that  will  comprise  each  subsystem  or  component.  In  this  way,  the 
physical  architecture  includes  a  representation  of  the  software  and  hardware 
“components”  necessary  to  realize  the  system.  This  physical  view  then  provides  a  clear 
definition  of  the  elements  of  the  system  that  are  needed  to  perform  the  SRL  assessment. 

In  brief,  a  system  architecture  model  is  needed  to  reason  about  a  problem  in  a  scientific 
way.  So  to  find  the  bottlenecks  and  "imperfections"  of  a  system,  there  is  a  need  for  a 
consice  but  mighty  method  of  modeling  a  system  of  interest.  DoDAF  provides  those 
models  in  terms  of  many  models  and  views.  The  DoD  Architecture  Framework 
(DoDAF)  defines  a  common  approach  for  DoD  architecture  description,  development, 
presentation,  and  integration.  DoDAF  version  2.0  describes  four  related  technical 
viewpoints  of  architecture: 

1.  The  Capability  Viewpoint  (CV)  identifies  the  requirements  and  delivery  of  the 
system. 

2.  The  Operational  Viewpoint  (OV)  identifies  what  needs  to  be  accomplished  and 
who  does  it. 

3.  The  Services  Viewpoint  (SvcV)  identifies  the  services  and  exchanges  in  a  services 
oriented  architecture. 

4.  The  Systems  Viewpoint  (SV)  relates  systems  and  characteristics  to  operational 
needs. 
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Products  within  this  framework  can  be  associated  with  the  physical  architecture.  The 
SvcV  and  the  SV  are  very  closely  related.  Indeed,  they  could  be  considered  to  be  two 
alternate  views  of  the  design/implementation  of  a  system. 

The  SV  represents  the  traditional  physical  design  of  a  system,  where  as  the  SvcV 
represents  a  more  modern  Service  Oriented  Architecture  (SOA)  design  of  a  system.  In 
this  manner,  the  SV  captures  the  information  on  supporting  automated  systems, 
interconnectivity,  and  other  systems  functionality  in  support  of  operating  activities.  As 
SOA  becomes  more  predominant  it  can  be  expected  that  over  time,  the  DoD’s  emphasis 
on  Service  Oriented  Environment  and  Cloud  Computing  may  result  in  the  elimination  of 
the  SV.  Therefore  either  the  SV-i  or  the  SvcV-i  can  be  considered  the  physical  design  of 
the  system  depending  on  the  architectural  approach  taken  by  the  design  team.  These 
views  can  be  considered  as  complementary  design  views. 

Physical  description  is  captured  in  several  DoDAF  products  in  either  the  SV  or  the  SvcV 
model: 


•  SV(SvcV)-i,  Systems  Descriptions 

With  further  descriptions  to  be  found  in 

•  SV(SvcV)-2,  Systems  Flow  Descriptions 

•  SV(SvcV)-3,  Systems(Services)-Systems(Services)  Matrix 

•  SV(SvcV)-7,  Systems(Services)  Measures  Matrix 

The  elements  of  the  target  environment  can  be  established  by  examination  of  the 
physical  view  of  the  system  design.  If  the  program  is  using  a  DoDAF  structure,  this  can 
be  accomplished  by  examination  of  the  SV-i  or  SvcV-i  diagram.  The  components  blocks 
on  the  diagram  represent  the  elements  (components)  of  the  system.  These  are  the 
candidate  items  to  be  reviewed  for  possible  use  in  the  SRL  assessment.  All  of  the 
connectors  on  the  diagram  represent  interfaces  that  will  be  candidates  for  evaluation  as 
part  of  the  SRL. 

To  increase  the  understanding  of  all  system  elements,  the  components  included  in  the 
derivative  models  SV(SvcV)-2,  SV(SvcV)-3,  and  SV(SvcV)-7  can  be  checked  to  see  that 
all  required  elements  have,  in  fact,  been  identified.  Another  check  can  be  performed  by 
examining  the  program  WBS  and  comparing  the  hierarchical  development  tasks  against 
the  elements  defined  for  the  SRL  methodology.  The  result  of  this  effort  should  be  a 
definition  of  all  the  elements  and  interfaces  of  the  system  that  is  very  close  to  the 
program  design  and  development  work  definition. 

The  remainder  of  this  section  describes  the  process  for  defining  all  the  architectural 
elements  to  be  modeled  in  the  SRL.  This  can  be  performed  in  five  steps: 

l.  Analyze  the  architecture  to  determine  all  the  elements  in  the  system 
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2.  Identify  the  Critical  Technology  Elements  (CTEs).  These  will  be  evaluated  and 
scored  in  the  SRL. 

3.  Identity  the  Non-critical  Technology  Elements  (NTEs).  These  will  not  be 
evaluated  and  scored  in  the  SRL.  These  elements  will  not  impact  the  SMA 
Analysis. 

4.  Identify  the  Critical  Technology  Integrations  (CTIs).  These  will  be  evaluated  and 
scored  in  the  SRL. 

5.  Identify  the  Non-critical  Technology  Integrations  (NTIs).  These  will  not  be 
evaluated  and  scored  in  the  SRL. 

It  should  be  pointed  out  that  the  “system”  under  development  can  be  at  any  level  of 
decomposition.  The  level  of  decomposition  is  determined  by  the  program.  It  should  be 
at  a  level  of  decomposition  that  the  program  comfortably  feels  the  major  technology 
components  can  be  identified.  At  this  point  we  focus  on  the  physical  model  of  the 
system.  When  we  begin  to  work  in  the  next  step  we  will  use  the  functional  model,  and  in 
particular,  the  allocation  of  functional  to  physical.  The  next  step  in  the  SRL  architecture 
definition  process  is  to  identify  those  system  components  that  are  CTEs  in  the 
development  of  the  system.  These  will  be  the  elements  that  will  need  to  be  evaluated, 
rated,  and  compiled  in  the  SRL  assessment.  The  selection  of  these  CTE  components  is 
discussed  in  the  next  section  of  this  paper. 

Finally,  DoDAF  2.0  has  added  new  views  and  products,  in  what  has  been  described  as  a 
fit  for  purpose,  where  emphasis  is  shifted  to  the  data  and  artifacts,  rather  than  models. 
This  change  suits  the  prospects  of  system  maturity  assessment  using  system 
architectures,  as  criteria  and  information  harvested  in  a  system  architecture  can  be  used 
to  aid  decision  makers  when  it  comes  to  TRL  and  IRL.  Figures  3  and  4  are  extracted 
from  an  OV-i,  OV-2,  and  OV-5  product  of  a  DoDAF  model  which,  based  on  the  AV-i 
model,  explains  a  Coordinated  Land  and  Sea  attack.  Figure  3  is  captured  from  IBM 
Rational  Rhapsody,  and  this  is  from  a  DoDAF  example  where  a  coordinated  Land  and 
Sea  Attack  is  shown.  Given  this  high  level  description,  we  navigate  to  an  OV-2,  which 
indicates  the  key  players  and  the  interactions  necessary  to  conduct  the  corresponding 
operational  activities  of  OV-5a  or  OV-5b.  The  dialog  box  on  the  right  side  contains 
information  regarding  the  technology  that  can  be  used  for  maturity  assessment.  The 
two-column  dialog  box  is  highly  customizable;  a  click  on  edit  enables  highly 
customizable  information  to  be  added  or  allocated  to  support  assessment  of  a 
technology  for  a  certain  level.  Figure  3  shows  a  hypothetical  set  of  criteria  required  for 
the  evaluation  of  a  technology  in  the  TRL  range  of  3. 

Utilizing  the  features  of  OV-2  helps  us  gain  insight  into  the  integration  advances  of  the 
UAV  and  COMMS,  which  is  an  aircraft  and  satellite  depicted  above.  The  information 
provided  in  the  dialog  box  below  is  set  to  automatically  retrieve  the  latest  information, 
or  point  an  assessor  to  the  latest  data  locations  which  would  support  the  readiness  level 
assessment.  Figure  4  is  an  example  that  shows  the  hypothetical  criteria  required  for  the 
assessment  of  the  IRL  level  of  3. 
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Figure  4:  IBM  Rational  Rhapsody  DoDAF  Example  for  IRL  Criteria 


Contract  Number:  PI98230-08-D-01 71 


DO  002,  TO  0002,  RT0012 


Report  No.  SERC-2011-TR-014 
FINAL  January  21,  2011 

UNCLASSIFIED 

30 


UNCLASSIFIED 


4.2  Mapping  SRL/ITRL  to  Development  Activities/Status 

In  defining  a  mapping  of  the  SRL  to  developmental  activities/status  we  wanted  to  make 
sure  that  there  was  enough  flexibility  in  the  SRL  Method  to  allow  for  the  values  to  be 
correlated  or  adjusted  to  varying  organizational  practices.  What  we  will  describe  is  how 
it  was  implemented  within  PMS420  and  supported  by  Northrop  Grumman;  although, 
these  same  practices  have  been  used  in  other  organizations  within  Lockheed-Martin  and 
NASA. 

4.2.1  TRL/IRL  Assessment 

For  determining  the  TRL  of  a  technology,  there  are  methods  (e.g.  Missile  Defense 
Agency  Checklist),  processes  (e.g.  TRA  Deskbook),  and  tools  (e.g.  AFRL  TRL  Calculator; 
Department  of  Homeland  Security  (DHS)  TRL  Calculator).  Some  of  these  have  become 
common  practice  in  the  DoD,  but  there  is  limited  guidance  on  IRL.  In  conjunction  with 
the  nine  level  IRL  already  specified,  we  have  developed  supporting  guidance  that  further 
clarifies  each  IRL.  This  is  detailed  in  Appendix  A  and  further  explained  by  Sauser,  et  al. 
(2010). 


4.2.2  SRL/ITRL  Interpretation 

Aside  from  the  SRL  providing  an  assessment  of  overall  system  development,  in  concert 
with  the  ITRL  it  can  also  be  a  guide  in  prioritizing  potential  areas  that  require  further 
development.  That  is,  if  we  are  considering  a  “systems-focused  approach”  to  our 
methodology,  then  we  cannot  evaluate  a  system  based  on  just  a  single  number,  such  as 
the  SRL  value  alone.  As  illustrated  in  Figure  3,  the  ITRL,s  (technologies  with  their 
integration  links  considered)  present  a  spectrum  showing  some  technologies  with  their 
integrations  considered  whose  readiness  levels  could  be  in  different  development  phases 
other  than  the  overall  system.  While  it  could  be  argued  that  the  overall  SRL  is  only  as 
good  as  the  lowest  TRL  or  ITRL,  this  perspective  would  also  lose  sight  of  even  those 
technologies  that  are  potentially  developing  faster  than  the  system.  In  understanding 
the  value  of  the  SRL  analysis,  we  must  understand  the  spectrum  of  ITRL  and  its 
relationship  to  the  SRL.  In  Figures  5  and  6,  we  see  the  same  overall  system  providing  a 
functionality  and  capability  but  with  different  technology  and  integration  options.  If  we 
focused  on  the  SRL  alone  in  comparing  these  two  systems,  we  would  not  see  the 
influence  these  options  have  on  the  systems  development  because  we  would  only  see  a 
0.04  difference  in  the  SRL  (0.60  to  0.64).  In  effectively  utilizing  the  SRL  method,  you 
have  to  consider  all  of  the  data,  i.e.  SRL  and  ITRLs,  and  use  this  as  better  insight  into 
system  maturity. 
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Figure  5:  SRL/ITRL  Reporting  (provided  by  Northrop  Grumman) 
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Figure  6:  SRL/ITRL  Reporting  (provided  by  Northrop  Grumman) 
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4.2.3  SRL  Mapping 

Despite  the  utility  of  calculating  a  SRL  and  its  supporting  ITRLs,  without  an  articulated 
correlation  to  qualitative  systems  engineering  practices,  it  becomes  difficult  to 
determine  the  added  value  in  understanding  its  implication  on  the  development 
lifecycle.  To  address  this  we  have  subsequently  performed  further  verification  and 
validation  of  this  approach  to  other  cases  in  conjunction  with  system  developers  from 
DoD,  Lockheed-Martin,  NASA,  and  Northrop  Grumman.  This  work  continues  and  will 
need  extensive  validation  to  provide  the  level  of  confidence  needed  in  its  practice  to 
make  minimal  risk  decisions.  In  general,  the  0.01.0  SRL/ITRL  range  that  allows  status 
to  be  efficiently  mapped  to  program  development  maturity  is  represented  in  Figure  7. 
While  this  mapping  indicates  distinct  numeric  values  at  phase  transition  points,  in 
practice,  we  contend  that  these  transitions  should  be  managed  within  a  tolerance  of  risk 
acceptance.  For  example,  a  SRL/ITRL  for  Milestone  B  may  have  a  tolerance  of  +/- 10%, 
indicating  that  an  SRL/ITRL  value  that  falls  within  this  tolerance  is  still  acceptable  for 
progressing  past  Milestone  A. 
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Figure  7:  SRL  and  ITRL  Mapped  Against  DoD  Defense  Acquisition  Lifecycle 


4.2.4  SRL  Reporting 

In  our  development  of  an  SRL  method,  we  strived  to  maintain  a  systems-focused 
approach  that  would  create  a  metric(s)  to  address  some  of  the  current  concerns  with  the 
TRL.  What  resulted  was  a  set  of  metrics  and  an  approach  that  can  have  the  following 
implications  on  defense  acquisition: 

■  The  SRL,  ITRL,  IRL,  and  TRL  provide  an  enhanced  capability  alignment  through 
the  identification  of  specific  technology,  integration,  and  system  maturities  that 
can  be  used  as  a  trade-study  tool  to  select  the  most  appropriate  technologies  and 
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integrations  to  obtain  the  lowest  amount  of  risk,  cost,  and  time  and  satisfy  a  given 
customer  need.  This  can  be  observed  by  comparing  Figures  5  and  6. 

■  The  SRL  Method  can  improve  customer  confidence  in  the  acquisition  manager  by 
providing  a  qualification  of  system  maturity  in  relation  to  system  functionality.  It 
can  also  provide  improved  understanding  of  the  system’s  mission  capabilities  in 
terms  of  readiness  criteria. 

■  The  SRL/ITRL  can  provide  an  assessment  of  maturity  at  multiple  architectural 
layers.  Any  single  SRL  assessment  contains  multiple  ITRL  assessments,  which 
can  provide  insight  into  the  interdependencies  of  different  sub-functions  and  how 
they  fit  within  the  larger  architecture.  This  can  be  observed  by  comparing  Figures 
5  and  6. 

■  The  SRL,  IRL,  and  TRL  provide  common  ontology  to  measure  and  describe 
acquisition  development,  system  development  and  technology-insertion 
evaluation. 

■  The  IRL  reduces  the  uncertainty  involved  in  integrating  a  technology  into  a 
system  and  identifies  integration  as  a  separate,  specific  metric. 

It  also  becomes  important  in  how  the  SRL/ITRL  results  are  reported.  Figures  5  and  6 
are  examples  provided  by  Northrop  Grumman  in  how  they  report  SRL/ITRL  to  the 
PMS420,  but  also  SRL  data  can  be  reported  via  planned  milestones  as  shown  in  Figure  8 
(also  provided  by  Northrop  Grumman). 
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Figure  8:  SRL  Planned  versus  Actual  Reporting  Example  (provided  by  Northrop  Grumman) 


4.3  Leveraging  SRL  Relationships  to  Allocate  Resources 

SRL  was  first  described  in  a  cost  model  by  Sauser  and  Ramirez-Marquez  (2009)  to 
provide  information  about  which  technologies  and  integrations  to  advance  to  which 
readiness  level  such  that  the  maturity  of  the  system  is  maximized  based  on  limited 
resources  made  available  for  system  development.  In  this  section  we  will  describe  this 
optimization  model  as  it  is  applied  to  the  development  of  a  system  to  illustrate  how  SRL 
can  be  used  to  plan  development.  As  an  example,  the  systems  engineer  or  program 
manager  who  is  concerned  with  utilizing  resources  allocated  for  the  system  can  now  set 
development  goals  such  that  the  maximum  amount  of  system  readiness  is  achieved.  In 
order  to  execute  the  development  required  to  have  maximum  SRL  value,  it  is  necessary 
to  know  how  to  utilize  the  resources  optimally.  That  is,  the  systems  engineering  or 
program  manager  must  determine  which  of  the  system  components  should  be  matured 
to  what  levels  so  they  can  allocate  the  available  resources  accordingly.  We  have 
previously  articulated  this  model  in  the  form  of  a  resource  allocation  of  budget  and 
schedule;  described  in  detail  in  (Sauser  and  Ramirez-Marquez,  2009).  For  this  research, 
we  expanded  upon  this  model  to  utilize  the  construct  of  Equivalent  System  Mass  (ESM). 
Thus,  the  general  mathematical  form  of  this  model  follows: 
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Model  ESM  SRL  max 
Maximize:  SRL  (TRL,  IRL) 

Subject  to:  ESM(TRL,IRL)  <  esm 


4.3.1  Equivalent  System  Mass 

ESM  was  first  defined  in  1997  as  a  metric  for  comparing  technology  options  for  the 
space  life  support  systems  (Drysdale,  2003).  It  allowed  for  the  tradeoff  of  mass,  volume, 
power,  cooling  and  crew  time  based  on  a  single  mass  value.  The  fundamental  premise 
was  that  a  mass  value  could  be  equated  to  launch  cost  (e.g.  it  costs  $10,000  per  pound 
to  launch  a  payload  on  the  Space  Shuttle),  thus  allowing  for  the  optimization  of 
technology  options  to  achieve  mission  objectives.  It  is  a  common  practice  in  space 
systems  development  for  mass,  as  it  relates  to  cost,  to  be  a  driver  in  determining  the 
deployment  success  of  space  products  (Saleh  et  al.,  2007,  Koelle,  2005,  Carli  and  Pablos, 
2006).  Although  cost  as  an  independent  variable  in  the  design  of  space  systems  has 
been  prevalent  throughout  the  industry  for  decades,  there  is  a  need  to  shift  the  emphasis 
of  cost  as  a  driver  in  the  analysis  for  engineering  space  systems  (Saleh  et  al.,  2007).  This 
was  a  fundamental  motivation  for  using  ESM  in  lieu  of  dollar  costs  for  technology 
development  (Drysdale,  2003).  ESM  allows  for  cost  to  become  an  independent  variable 
and  does  not  have  a  direct  influence  on  a  trade  analysis.  In  summary,  ESM  provides 
advantages  since  cost  estimates: 

•  can  be  politically  sensitive; 

•  are  not  generally  released; 

•  do  not  always  include  all  cost;  and 

•  tend  to  be  complex  and  dynamic  (Drysdale,  2003). 

Accordingly,  ESM  is  calculated  as: 


ESM  =  M +L  +  V*  eqV  +  P*eqP  +  C*  eqC+CT*  D*  eqCT 


where:  ESM  =  equivalent  system  mass  value  of  the  system  of  interest  [kg],  M  =  total 
mass  of  the  system  [kg],  L  =  mass  of  the  materials  and  spare  logistics  of  the  system  [kg], 
V  =  total  pressurized  volume  of  system  [m3],  eqV  =  mass  equivalency  factor  for  the 
pressurized  volume  infrastructure  [kg/m°»],  P  =  total  power  requirement  of  the  system 
[kWe],  eqP  =  mass  equivalency  factor  for  the  power  generation  infrastructure  [kg/kWe], 
C  =  total  cooling  requirement  of  the  system  [kWth],  eqC  =  mass  equivalency  factor  for 
the  cooling  infrastructure  [kg/kWth],  CT  =  total  crew  time  requirement  for  operation 
and  maintenance  of  the  system  [CM-h/day],  D  =  duration  of  the  mission  segment  of 
interest  [day],  and  eqCT  =  mass  equivalency  factor  for  the  crew  time  support  [kg/CM- 
h].  For  a  detailed  explanation  and  guidance  on  ESM  see  (Levri  et  al.,  2003). 
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ESM  was  investigated  for  this  research  as  mass  can  be  a  driver  in  the  development  and 
potential  deployment  of  mission  modules  as  they  relate  to  the  efforts  of  PMS420  and 
other  Littoral  Combat  Ship  efforts.  While  ESM  adds  value  to  the  trade  analysis  of 
technology  options  for  space  missions,  it  has  been  noted  in  those  efforts  of  space 
systems  that  it  should  not  be  a  standalone  metric  and  additional  metrics  that  evaluate 
the  developmental  status  of  a  technology  would  be  of  added  value.  More  commonly, 
TRL,  as  a  measure  of  technology  maturity,  has  been  repeatedly  cited  as  a  core  metric 
that  should  be  used  with  ESM  (Rodriquez  et  al.,  2004,  Russell  and  Carrasquillo,  2007, 
Drysdale,  2003,  Czupalla  et  al.,  2004).  To  further  enhance  the  use  of  TRL  with  ESM,  we 
created  an  SRLmax  model  that  utilized  ESM  with  SRL,  i.e.  ESM_SRLmax. 


4.3.2  Equivalent  Systems  Mass  Optimization  (ESM_SRLMAx) 

The  matrices  IRL  and  TRL  of  the  model  contain  the  decision  variables.  Each  of  these 
variables  is  integer  valued  and  bounded  by  ( IRLij,g )  and  (TRL,, 9),  respectively.  That  is, 
the  TRL/IRL  for  any  component  cannot  be  below  its  current  level  or  above  perfect 
technology  or  integration  development  (IRL  or  TRL  =  9).  The  objective  function  of 
Model  ESM_SRLmax  of  the  system  is  a  function  of  the  decision  variables,  which  dictate 
how  the  different  levels  for  both  TRL  and  IRL  are  improved.  The  left  hand  side  of  the 
inequality  constraint  represents  the  ESM  as  a  function  of  the  improved  TRLs  and  IRLs, 
and  the  right  hand  side  indicates  the  total  amount  allowance  of  ESM  for  the  whole 
system.  Since  the  ESM  is  an  indicator  of  the  needed  launch  cost,  the  model  tries  to 
maximize  the  system  maturity  while  remaining  under  the  ESM  allowance,  thus  meeting 
the  cost  constraint. 

To  completely  characterize  the  decision  variables,  it  is  necessary  to  introduce  the 
following  transformation: 

yk=  1  If  TRLt  =  k  and  xk  =  1  if iRLt]=k  for/c=i,...9 
0  otherwise  0  otherwise 

Notice  that  based  on  these  binary  variables,  each  of  the  possible  normalized  TRL  and 

«  9 

ky-  fa* 

TRL  =  k=l  IRL  =  k=l 

IRL  in  the  system  can  be  obtained  as:  '  9  and  "  9 

and  ITRL  is  transformed  to: 

99  99  99  99  n99 

fccfj-r-  ky[+  fcc*2-~  kxk+  ky kx*n+  kykn+  fag*  ty)* 

TTDI - 4=1  4=1  4=1  4=1  4=1  4=1  .  4=1  4=1  _  7=14=1 _ 4=1 _ 


The  model  belongs  to  the  class  of  binary,  integer-valued,  non-linear  problems.  For 
example,  a  system  with  6  technologies  containing  10  distinct  integrations,  and  assuming 
all  technologies  and  integrations  are  at  their  lowest  levels,  there  can  be  as  many  as  96+w 
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potential  solutions  to  the  model.  Evaluating  each  possible  solution  is  prohibitive,  so  to 
generate  a  more  timely  optimal  solution,  a  meta-heuristic  approach  developed  by 
Ramirez-Marquez  and  Rocco  (2008)  is  applied  to  solve  the  optimization  model.  This 
approach,  called  Probabilistic  Solution  Discovery  Algorithm  (PSDA),  has  the  capability 
of  producing  quasi-optimal  solutions  in  a  relatively  short  period  of  time.  However,  it 
must  be  mentioned  that  the  results  cannot  be  proven  to  be  the  optimal  solution.  This  is 
because  by  taking  a  probabilistic  approach,  the  algorithm  can  only  select  subsets  of  the 
entire  feasible  set  from  which  to  find  a  solution.  Every  time  the  algorithm  is  run,  a 
different  subset  is  selected.  Nevertheless,  prior  tests  have  indicated  that  PSDA  results 
tend  to  be  better  than  results  from  alternative  meta-heuristic  approaches  (Ramirez- 
Marquez  and  Rocco,  2007). 

As  used  in  the  solution  of  the  maximization  problem,  after  the  algorithm  is  initialized,  it 
follows  three  inter-related  steps: 

•  Strategy  Development  -  a  Monte  Carlo  simulation  is  used  to  identify  to  what 
potential  TRL  or  IRL  levels  the  technologies  and  links  can  be  advanced  or  matured; 

•  Analysis  -  each  potential  solution  is  analyzed  by  calculating  its  associated  SRL  and 
ESM; 

•  Selection  -  through  an  evolutionary  optimization  technique,  a  new  optimal  set  of 
technologies  and  integration  links  (with  their  corresponding  TRLs  and  IRLs  is  chosen 
(based  on  the  SRL  and  ESM  values). 

During  Strategy  Development,  based  on  the  probabilities  defined  by  vectors  y;u  and  y iju, 
the  simulation  is  used  to  generate  a  specified  number  (defined  by  V)  of  potential 

v  v  ^ 

designs,  TRI  and  IRI  (v=i,..,V).  For  each  technology  i,  iu  (the  kth  element  of  vector  y;u) 
defines  the  probability  that  at  cycle  u,  the  TRL  of  such  a  technology  will  increase  its 

k 

current  readiness  to  level  k  (i.e.  *  =  p(yk!  =  1)).  Similarly,  m  defines  the  probability  that  at 
cycle  u,  the  IRL  between  the  ith  and  jth  technologies  will  increase  its  actual  readiness  to 

level  k  (i.e.  IJU  ~P(X‘J -1)).  This  step  also  contains  the  stopping  rules  of  the  algorithm.  In 
essence,  the  first  rule,  which  is  used  in  this  paper,  allows  the  user  to  set  a  specific 
number  of  cycles.  The  second  rule  dictates  the  algorithm  to  be  stopped  once  both  vector 
y iu  and  jiju  can  no  longer  be  updated  (i.e.  all  initial  “appearance”  probabilities  are  either 
zero  or  one).  In  the  context  of  this  algorithm  a  cycle  is  understood  as  every  time  the 
value  u  is  updated. 

The  second  step,  Solution  Analysis,  implements  the  approach  discussed  in  Sauser  et  al. 
(2008a)  and  previously  summarized  to  obtain  the  SRL,  and  the  ESM  of  the  development 
associated  with  each  of  the  potential  system  design,  TRI  and  IRI  .  The  final  step  in  the 
algorithm  penalizes  the  SRL  of  the  potential  designs  generated  in  cycle  u  whenever  they 
violate  the  ESM  constraints.  The  solutions  are  then  ranked  in  decreasing  order  of 
magnitude  with  respect  to  the  penalized  SRL.  Then,  the  best  of  these  solutions  is  stored 
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in  set  K  and  finally,  a  subset  of  size  S  of  the  ranked  feasible  solutions,  is  used  to  update 
the  probabilities  defined  by  the  vectors  y iu  and  yi;u.  These  new  vectors  are  re-evaluated  in 
Step  l  to  check  for  termination  or  for  solution  discovery.  Finally,  when  the  prescribed 
number  of  cycles  has  been  reached,  the  best  solution  in  set  K  is  chosen  as  the  optimal 
system  design. 


4.3.3  Example:  Results  and  Discussion 

There  can  be  several  technology  options  proposed  to  address  any  system  solution.  Each 
of  the  technology  options  presents  various  design  and  development  challenges,  e.g. 
maturity,  mass,  budget.  For  our  simulation  we  will  run  the  ESM_SRLmax,  demonstrating 
the  insights  that  the  addition  of  SRL  to  the  calculation  of  ESM  can  have  on  making  more 
informed  development  decisions.  We  will  demonstrate  this  using  two  nonspecific 
systems  with  variations  in  three  technology  options  (i.e.  Technologies  3,  4,  and  6)  and 
thus  their  ESM.  These  systems  have  six  technologies  containing  ten  distinct 
integrations;  we  will  call  them  System  X  and  Systems  Y.  We  could  think  of  these 
systems  as  delivering  the  same  capability  but  with  different  technology  options.  Tables 
3  and  4  show  the  TRL  and  IRL  values  and  Tables  5  and  6  show  the  ESM  values  of 
Systems  X  and  Y. 


Table  3:  Readiness  Levels  of  System  X 


Technology 

TRL 

Technology  l 

5 

Technology  2 

4 

Technology  3 

5 

Technology  4 

4 

Technology  5 

5 

Technology  6 

6 

Integration 

IRL 

Integration 

IRL 

1,2 

4 

2,6 

4 

1,5 

5 

3,4 

4 

2,3 

4 

3,5 

5 

2,4 

4 

4,6 

6 

2,5 

4 

5,6 

5 

Table  4:  Readiness  Levels  for  Systems  Y 


Technology 

TRL 

Technology  1 

5 

Technology  2 

4 

Technology  3 

7 

Technology  4 

5 

Technology  5 

5 

Technology  6 

8 

Integration 

IRL 

Integration 

IRL 

1,2 

4 

2,6 

4 

1,5 

5 

3,4 

4 

2,3 

4 

3,5 

5 

2,4 

4 

4,6 

6 

2,5 

4 

5,6 

5 
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Based  on  the  current  readiness  levels  of  its  technologies  and  integration  links  as  shown 
in  Tables  3  and  4,  System  X  would  yield  a  SRL  of  0.39,  and  System  Y  would  yield  a  SRL 
of  0.33.  Referring  to  Figure  7,  these  values  indicate  that  both  of  these  system  scenarios 
should  be  in  Phase  A:  Concept  &  Technology  Development. 

Tables  5,  6,  and  7  show  the  ESM  of  each  component  (technology  or  integration)  at 
different  maturity  levels.  For  example,  to  mature  Technology  1  from  TRL  of  1  to  9 
Systems  X s  ESM  is  estimated  to  rise  from  2,743  to  3,234  kgs.  At  their  current  TRLs  and 
IRLs,  the  overall  ESM  for  System  X  is  43,273  and  59,079  for  System  Y.  In  order  to  fully 
mature  all  the  technology  and  integration  elements,  System  X  is  allowed  a  maximum 
ESM  of  44,876  and  System  Y  of  60,122  without  any  budgetary  tolerances.  These  values 
are  obtained  as  the  sum  of  the  ESM  values  when  all  TRLs  and  IRLs  are  equal  to  9. 


Table  5:  Cumulative  ESM  for  Technology  Elements  against  TRL  (Current  TRLs  in  Bold)  -  System  X 


TRL 

Technology 

1 

2 

3 

4 

5 

6 

1 

2743 

2302 

3350 

1302 

2926 

17139 

2 

2835 

2551 

3489 

1385 

3074 

18499 

3 

2986 

2633 

3765 

1389 

3273 

19778 

4 

3058 

2767 

3897 

1462 

3356 

19864 

5 

3131 

2836 

3926 

1498 

3476 

20466 

6 

3212 

2873 

4004 

1510 

3526 

20988 

7 

3230 

2898 

4044 

1521 

3562 

21357 

8 

3233 

2907 

4096 

1536 

3580 

21521 

9 

3234 

2911 

4111 

1538 

3597 

21610 

Table  6:  Cumulative  ESM  for  Technology  Elements  against  TRL  (Current  TRLs  in  Bold)  -  System  Y 


TRL 

Technology 

1 

2 

3 

4 

5 

6 

1 

2743 

2302 

6413 

1956 

2926 

25635 

2 

2835 

2551 

6679 

2081 

3074 

27669 

3 

2986 

2633 

7208 

2087 

3273 

29582 

4 

3058 

2767 

7460 

2197 

3356 

29711 

5 

3131 

2836 

7516 

2251 

3476 

30611 

6 

3212 

2873 

7665 

2269 

3526 

31392 

7 

3230 

2898 

7742 

2285 

3562 

31944 

8 

3233 

2907 

7841 

2308 

3580 

32189 

9 

3234 

2911 

7870 

2311 

3597 

32322 
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Table  7:  Cumulative  ESM  for  Integration  Elements  against  IRL  (current  IRLs  in  bold) 


IRL 

Integration 

1,2 

1,5 

2,3 

2,4 

2,5 

2,6 

3,4 

3,5 

4,6 

5,6 

1 

624 

963 

1352 

371 

689 

757 

703 

241 

279 

543 

2 

679 

1017 

1477 

395 

701 

846 

765 

260 

294 

547 

3 

693 

1088 

1514 

417 

729 

896 

805 

276 

296 

585 

4 

729 

1090 

1540 

431 

744 

943 

847 

279 

300 

589 

5 

749 

1092 

1565 

438 

763 

956 

881 

290 

302 

604 

6 

761 

1116 

1581 

441 

773 

972 

901 

293 

303 

608 

7 

770 

1130 

1597 

442 

778 

973 

905 

294 

308 

612 

8 

776 

1136 

1600 

446 

784 

978 

908 

297 

310 

613 

9 

779 

1144 

1601 

448 

787 

979 

914 

299 

312 

614 

To  further  explain  the  model,  we  will  describe  a  situation  where,  for  example,  the 
program  manager  wants  to  show  the  customer  to  which  maturity  level  or  development 
stage  they  can  take  a  system  if  they  are  given  various  ESM  allowances.  For  simplicity  we 
will  focus  on  System  X  and  then  compare  the  results  of  the  two  systems.  In  order  to 
answer  this,  the  PSDA  optimization  model  computed  the  respective  maximum  SRL 
values  when  20%,  40%,  60%,  80%  and  all  of  the  ESM  allowance  is  allocated.  The 
results  are  shown  in  Table  8.  For  example,  when  the  ESM  is  allowed  to  increase  from 
43,273  (current  value)  to  43,901  (utilizing  around  40%  of  the  remaining  allowable 
increase  in  ESM),  the  SRL  can  be  increased  from  0.33  to  0.76.  This  takes  System  X 
from  Phase  A  to  a  state  where  it  would  soon  transition  from  Phase  B:  Preliminary 
Design  &  Technology  Completion  to  Phase  C:  Final  Design  &  Fabrication.  In  addition, 
the  development  plan  which  can  achieve  the  SRL  value  of  0.76  when  40%  of  the 
incremental  ESM  is  allocated  also  shows  that  the  subsystems  which  are  based  on  each 
technology  element  reach  their  respective  maturity  levels  as  shown  in  Table  8.  The  40% 
case  shows  that  of  the  six  subsystems,  three  are  ahead  (ITRLi,4,6),  two  are  slightly  behind 
(ITRL2,5)  and  one,  (ITRL3)  is  close  to  the  same  level  as  that  of  the  whole  system.  This 
insight  can  become  useful  when  the  maturity  levels  are  associated  with  systems 
engineering  activities;  hence,  the  spectrum  of  ITRLfs  can  indicate  levels  of  variation  in 
the  systems  engineering  activities  which  are  needed  to  mature  the  entire  system. 


While  the  SRL  scale  can  have  value  for  overall  planning,  one  can  assess  the 
developmental  maturity  of  each  technology  and  corresponding  integrations  based  on  the 
ESM  allowances  using  Model  ESM_SRLmax.  Table  9  illustrates  the  associated  TRL  and 
IRL  levels  obtained  from  the  optimal  solution  for  each  of  the  cases  considered.  This  is 
very  important  to  understanding  how  the  optimization  approach  can  influence  the 
developmental  maturity  of  the  individual  technologies  and  integrations.  That  is,  the 
optimal  TRL  and  IRL  levels  obtained  from  the  model  becomes  a  guidance  tool  for  the 
systems  engineering  manager  to  better  understand  how  the  ESM  allowances  are 
impacting  maturity  of  development.  Table  9  also  indicates  that  for  ALS,  an  80%  ESM 
allowance  still  would  not  ensure  a  fully  mature  system  because  Technology  6  and  two  of 
the  IRLs  (2,3;  2,6)  are  not  completely  matured.  The  technology  involved  is  the  food 
processing  component  and  the  integration  elements  are  the  ones  that  connect  the  crew 
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habitat  to  it  as  well  as  to  the  water  processing  facility.  Unless  these  can  be  feasibly 
matured  in  space,  the  system  cannot  be  launched. 

It  must  be  pointed  out  that  the  design  solutions  in  Table  9  are  calculated  using  the 
budgeted  incremental  ESM.  However,  the  solution  for  each  increasing  amount  of 
allocated  ESM  is  not  dependent  on  the  values  of  the  readiness  levels  calculated  for  the 
preceding  lower  amount  of  ESM  allocation.  That  is,  the  algorithm  does  not  go 
sequentially  from  20%  to  40%  and  so  on,  such  that  20%  automatically  corresponds  to 
year  1  and  40%  to  year  2.  Rather,  what  the  solution  shows  is  that  if  a  certain  %  is 
allocated,  the  corresponding  technologies  and  integrations  can  be  matured  to  such 
levels  as  indicated.  It  is  up  to  the  decision  makers  to  allocate  a  budget  for  any  given  year 
and  plan  the  development  based  on  the  available  budget.  This  is  the  reason  why 
Technology  4  can  be  matured  to  level  9  under  20%  and  40%  ESM  allowance,  whereas  it 
is  only  matured  to  level  8  under  60%  ESM  allowance.  However,  if  a  time-related 
sequential  design  solution  is  desired,  say  for  5  years,  a  sequential  orderly  solution  can  be 
achieved  by  following  a  recursive  manner  of  utilizing  the  ESM_SRLmax  model.  For 
example,  in  order  to  get  an  incremental  design  solution  for  20%  and  40%  ESM 
allowances  corresponding  to  years  1  and  2  respectively,  first  execute  the  model  to  get  the 
design  solution  for  the  20%  scenario  then,  allocate  another  20%  for  year  2  and  re-run 
the  model.  That  is,  when  a  TRL  or  IRL  has  already  been  achieved  for  a  particular 
element,  it  can  no  longer  be  de-matured  just  to  follow  the  prescribed  solution  from  the 
algorithm.  Thus,  for  the  60%  scenario,  Technology  4  must  stay  at  TRL  9  and  not  revert 
back  to  8  as  a  practical  matter. 

When  we  compare  System  X  and  Y  we  can  see  the  insights  of  adding  the  SRL  with  ESM 
in  making  more  informed  development  decisions  (see  Tables  8-11).  The  comparison  of 
the  best  solutions  with  ESM  allowance  for  the  ITRL  and  SRL  (Tables  8  and  10)  of  the 
two  scenarios  does  show  as  much  significance  as  does  the  best  design  solution  for  every 
increase  in  ESM  allowance  (Tables  9  and  11).  By  comparing  the  design  solutions,  we  can 
observe  noticeable  variation  even  in  technologies  and  integrations  that  were  not  directly 
related  to  the  varying  technologies  options  (Technologies  3,  4,  and  6),  which  signify  the 
interrelationship  among  the  technologies.  For  example,  though  we  kept  the  IRLs 
constant,  the  recommended  design  solution  for  the  integration  between  technology  land 
2  is  noticeably  different.  While  we  kept  the  IRL  estimates  constant  in  the  two  scenarios, 
integration  is  where  we  are  observing  the  most  variance  in  design  solution.  Therefore, 
while  the  ESM  of  System  Y  is  much  higher  than  that  of  System  X,  there  may  be  other 
confounding  factors  that  influence  the  selection  of  technology  options  related  to 
technology  and  integration  maturity  and  their  relationship  to  ESM. 


Table  8:  Best  Solution  for  ESM  Increase  Allowance  -  System  X 


Case 

ITRLi 

itrl2 

ITRL3 

itrl4 

itrl5 

ITRL,, 

SRL 

ESM 

Current 

0-35 

0.28 

0.31 

0.33 

0.35 

0-37 

0-33 

43273 

20% 

O.50 

0.46 

0.47 

0.56 

0.50 

0.68 

0-53 

43579 

40% 

O.81 

0.69 

0.75 

0.79 

0.68 

0.83 

0.76 

43901 

60% 

0.96 

0.78 

0.89 

0.81 

0.85 

0.81 

0.85 

44221 

80% 

1.00 

0.90 

0.97 

0.92 

0.93 

0.86 

0-93 

44249 

100% 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

44876 
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Table  9:  Best  Design  Solution  for  Every  Increase  in  ESM  Allowance  -  System  X 


ESM  Allowance 

Technology 

Integration 

1 

2 

3 

4 

3 

6 

1,2 

1,3 

2,3 

2,4 

2,5 

2,6 

3,4 

3,5 

4,6 

5,6 

Current 

5 

4 

5 

4 

5 

6 

4 

5 

4 

4 

4 

4 

4 

5 

6 

5 

20% 

5 

4 

5 

9 

7 

6 

7 

7 

4 

7 

4 

7 

4 

8 

9 

8 

40% 

5 

9 

5 

9 

9 

6 

9 

8 

5 

9 

7 

7 

8 

9 

9 

8 

60% 

8 

9 

9 

8 

9 

6 

9 

9 

7 

7 

7 

7 

8 

9 

9 

8 

80% 

9 

9 

9 

9 

9 

6 

9 

9 

8 

9 

9 

7 

9 

9 

9 

9 

100% 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

Table  10:  Best  Solution  for  ESM  Increase  Allowance  -  System  Y 


Case 

ITRLi 

itrl2 

itrl3 

itrl4 

itrl5 

URL,, 

SRL 

ESM 

Current 

0-35 

0.32 

0.38 

0.42 

O.4O 

0.44 

0-39 

59079 

20% 

0.42 

0.43 

0.55 

0.65 

0.58 

0.72 

0.56 

59279 

40% 

0-54 

0.57 

0.64 

0.71 

0.66 

0.87 

0.67 

59493 

60% 

O.81 

0.73 

0.83 

0.89 

0.78 

0.97 

0.84 

59705 

80% 

1.00 

0.94 

0.94 

0.92 

0.93 

0.97 

0-95 

59863 

100% 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

60122 

Table  11:  Best  Design  Solution  for  Every  Increase  in  ESM  Allowance  -  System  Y 


ESM  Allowance 

Technology 

Integration 

1 

2 

3 

4 

5 

6 

l?2 

1,5 

2,3 

2,4 

2,5 

2,6 

3,4 

3,5 

4,6 

5,6 

Current 

5 

4 

7 

5 

5 

8 

4 

5 

4 

4 

4 

4 

4 

5 

6 

5 

20% 

5 

4 

7 

9 

8 

8 

4 

5 

4 

7 

4 

4 

4 

8 

9 

8 

40% 

5 

7 

7 

9 

9 

8 

6 

5 

4 

7 

4 

7 

4 

9 

9 

9 

60% 

5 

9 

7 

9 

9 

8 

8 

9 

5 

8 

6 

9 

9 

9 

9 

9 

80% 

9 

9 

7 

9 

9 

8 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

100% 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

9 

4.4  TRL  and  IRL  Evaluation  Linkage  to  Performance  Measures 

In  the  procurement  and  management  of  the  Mission  Packages  (MP’s)  for  a  system,  the 
designated  program  office  and  manager,  such  as  the  PMS420  for  the  Littoral  Combat 
Ship  program,  requires  insight  into  the  progress  of  the  Development  Program  Offices’ 
(DPOs)  constituent  mission  systems  and  where  they  stand  in  terms  of  providing 
anticipated  performance,  especially  the  Key  Performance  Parameters  (KPPs)  of  the 
system.  These  insights  are  critical  for  requisite  oversight  and  to  manage  development 
risks.  However,  the  issue  is  how  program  managers  accurately  can  predict  the  ability  of 
the  system  to  satisfy  KPPs  while  development  and  integration  are  proceeding. 
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Previously,  DPOs  were  able  to  use  Technical  Performance  Measures  (e.g.,  Technology 
Readiness  Levels  [TRL])  to  monitor  the  developmental  status  of  specific  technologies. 
With  the  development  of  complex  multi-capability  systems,  such  as  the  LCS, 
understanding  the  status  of  technologies  are  no  longer  sufficient  for  managers  to  gain 
the  requisite  level  of  knowledge  on  the  extent  to  which  the  KPPs,  as  designated  by  the 
Capability  Development  Document,  can  be  accomplished  for  the  designated  system. 
Volkert  (2009)  has  pointed  out  the  compounding  reasons: 

1.  The  capabilities  from  the  individual  constituent  mission  systems  are  often  being 
modified  or  utilized  in  manners  different  than  that  specified  in  their  original 
requirement  set.  Thus,  their  known/predicted  performance  may  be  different 
when  used  in  a  MP  SoS. 


2.  The  constituent  mission  systems  (capabilities)  being  developed  by  the  DPOs  are, 
in  some  cases,  still  maturing.  This  impacts  the  ability  to  determine  KPP 
performance  in  two  ways; 

a.  It  drives  an  incremental  fielding  of  capabilities  by  PMS  420,  meaning  the 
solution  set  for  accomplishing  (full  or  partially)  a  KPP  will  vary  over  time. 

b.  The  capability  delivered  by  the  DPO  may  not  provide  the  amount  of 
performance  anticipated/predicted  by  PMS  420  due  to  developmental 
challenges  within  the  DPOs  program. 

3.  The  combination  of  capabilities  available  to  choose  from  means  that  the  usage 
and  contribution  of  an  individual  capability  to  the  performance  of  a  KPP  can  vary 
depending  upon  the  operational  employment  of  the  system  within  a  SoS. 

Therefore,  for  predicting  the  achieved  proportional  capability  in  a  complex  system, 
Volkert  proposed  an  approach.  Here  we  re-write  his  formula  with  minor  changes: 

^"c(  1,2,  n.  =  *  *-*><7(1, 2,  .  .  .  ) 


Where  ac(i,2,...)  represents  the  maximum  level  of  performance  capability  the  combination 
of  technology  packages  (1,  2, ...)  is  expected  to  meet/provide.  con  represents  the 
weighting  factor  representing  the  proportional  level  of  performance  expected  at  the 
maturity  stage  of  n.  Tc(i,2,...)n  thus  represents  the  achieved  performance  level  of  the 
capability  which  can  contribute  to  the  satisfaction  of  the  KPP.  The  value  of  a  would  be 
expressed  in  the  units  of  performance  defined  by  the  KPP  while  co  would  be  unit  less. 

For  con,  Volkert  suggested  the  use  of  SRL  for  the  capability  at  that  time.  In  order  to 
differentiate  this  with  the  original  SRL  definition  that  is  designed  for  assessing  the 
development  maturity  of  the  whole  system,  we  introduce  the  new  notion  of  a  Capability 
System  Readiness  Level  (SRL_C)  to  measure  con  which  represents  the  readiness  of  the 
Capability  comprised  by  a  specific  combination  of  technologies  and  the  integrations 
among  them.  For  simplicity,  for  the  rest  of  this  section,  whenever  SRL  is  mentioned,  it 
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means  the  SRL_C.  Mathematically,  the  procedure  for  calculating  the  SRL_C  is  the  same 
as  with  SRL  but  considers  the  subset  of  n  technologies  from  within  the  system  which 
will  have  to  be  integrated  to  deliver  a  capability  C. 


4.4.1  Component  Importance  Measures 

A  system  has  variants  in  its  physical  architecture  that  realize  certain  functionality  and 
capability.  A  systems  engineer  or  acquisition  manager  would  make  trade-off  decisions  to 
find  a  solution  for  a  deployable  system.  Thus,  in  order  to  satisfy  Key  Performance 
Parameter  during  the  development  of  the  system,  this  paper  proposes  to  perform 
component  importance  analysis  by  introducing  three  Importance  Measures  (IMs)  with 
respect  to  SCS:  TRL/IRL,  cost,  and  labor-hours. 


4.4.2  SCS  WITH  RESPECT  TO  TRL/IRL  (lp) 

The  IM  of  TRL/IRL  evaluates  the  impact  of  a  change  in  the  development  maturity  of  an 
element  (i.e.  technology  or  integration)  on  system  development  maturity,  That  is,  IM 
measures  the  change  of  the  SRL  when  the  TRL  or  IRL  of  a  specific  element  changes 
from  its  current  value  to  a  target  value.  For  example,  let  srl(TRL,irl)  denote  the  current 

SRL  of  the  system,  and  SRIiTRL ,  IRL  \  TRLi  =  TRLj )  ( SRL(TRL,  IRL  \  IRLtj  =  IRL ) )  denote 

the  resultant  SRL  when  TRLi  ( IRLij )  changes  to  a  target  maturity  level  77?  L  (/7F)  and 

all  other  TRLs/IRLs  stay  on  current  maturity  values.  The  definition  of  IM  with  respect 
to  TRL/IRL  (F)  is  as  follows: 

For  TRL  I p  =  SRL(TRLJRL\TRL,  =TRL,) 

’  '  SRL(TRL,IRL ) 

p  SRL(TRL ,  IRL  I  IRL,  =  ffiL) 

For  IRIJ  = - - - - - - - * - L- 

1  SRL(TRL,IRL) 


Ip  implies  the  effect  of  change  in  the  readiness  level  of  a  given  component.  A  component 
for  which  the  variation  of  the  readiness  level  results  in  the  largest  variation  of  the  system 
SRL  has  the  highest  importance. 


4.4.3  SCS  WITH  RESPECT  TO  COST  (lCT) 

Zhang  et  al.  (2007)  suggests  that  classical  component  importance  analysis  ignores  cost, 
and  states  that  it  is  unrealistic  to  evaluate  the  importance  of  components  without 
considering  the  cost.  Hereby,  for  SRL  component  importance  analysis,  we  propose  to 
consider  the  economic  factor.  This  is  reasonable  by  noting  that  there  are  always 
situations  where  system  developers  have  to  make  the  investment  decisions  based  on  the 
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comparison  of  the  immediate  return  on  the  investment  of  dollars  needed  to  mature 
components.  Presumably,  especially  with  a  tight  budget,  developers  allocate  resources 
to  the  component  that  can  result  in  the  highest  system  maturity.  Therefore,  we  propose 
ICT  as  an  IM  that  takes  into  account  the  cost  for  maturing  components  to  facilitate  such 
comparisons.  Since  the  cost  to  mature  different  components  varies  and  improvements 
in  different  components  have  different  effects  on  SRL,  the  IM  that  takes  into  account  the 
development  cost  serves  as  a  baseline  to  compare  the  investment  returns  from  different 

_ CT 

components.  Let  '  TRL  /A/denote  the  associated  development  cost  for  maturing 

TRLi  from  its  current  readiness  level  to  a  target  level T  ‘ ,  and  ' J  1  ^  1  ^^denote 
the  associated  development  cost  from  maturing  IRLij  from  its  current  readiness  level  to 

a  target  level  IRLlj .  The  formula  to  calculate  the  ICT  is  as  follows: 


ForTRL 


CT  ASRL  SRL  (TRL,  IRL  \  TRLt  =  TRLt )  -  SRL(TRL,  IRL) 


CT: 


^TTRL;  CTj-RL, 


For  IRL ,l£T  = 


A SRL  SRL(TRL, IRL  \  IRL,  =  IRL ,  )  -  SRL(TRL, IRL) 


CT, 


ICT  implies  the  effect  of  the  cost  to  mature  a  given  component  on  SRL,  and  the 
component  whose  readiness  improvement  from  the  investment  results  in  the  largest 
gain  of  SRL  has  the  highest  importance. 


4.4.4  SCS  WITH  RESPECT  TO  LABOR-HOURS 

Besides  the  consideration  of  cost,  there  are  other  situations  (e.g.  when  only  certain 
labor-hours  are  available)  where  developers  care  more  about  the  return  on  the  effort 
needed  to  improve  components.  Therefore,  we  propose  another  importance  measure 
(ILH)  that  takes  into  account  the  associated  labor-hours  to  upgrade  the  component 

readiness  level  in  order  to  mature  the  SRL.  Let  ^ ^ F  > 

lh1  lh7 ^  lhtrl.(LH y-  lHjxtLj  iffjflii>\ienote  the  associated  development  labor-hours 

for  developing  TRLi  from  its  current  status  to  a  target  level  1 ,  and  1  ^ ' 1  /  A  ^  R  f 

^  J  denote  the  associated  development  labor-hours 

for  developing  IRLij  from  its  current  status  to  a  target  level  IRL>i ,  then  the  formula  for  ILH 
is  as  follows: 

ForTRL  ILH  -  ASRL  -  SRL  ( TRL  • IRL  I  TRLi  =  ™-i )  ~  SRL  ( TRL  > IRL ) 


LH; 


LHtrl~LHtrl, 


For  IRL  Llh  =  =  SRL{TRLJRL \IRL'J  =  MLj)-SRL(TRL,IRL) 

01  ’  ij  ~  LH,  ~ 


V 


L^IrF  ~  LH IRLij 
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ILH  implies  the  effect  of  the  labor-hours  or  effort  to  mature  a  given  component  on  SRL. 
The  component  whose  readiness  improvement  from  the  investment  of  labor-hours 
results  in  the  largest  gain  of  SRL  has  the  highest  importance. 


4.4.5  Example:  Results  and  Discussion 

The  following  example  was  used  in  (Forbes  et  al.,  2009)  to  illustrate  the  application  of 
SRL.  The  system  is  designed  to  perform  six  capabilities.  For  the  illustration  of  the 
proposed  methodology  in  this  paper,  it  is  assumed  that  the  mine-detection  capability 
that  is  enabled  by  the  combination  of  the  shaded  components  is  the  KPP  of  interest.  This 
capability  has  six  components  with  six  integrations  among  them,  and  the  corresponding 
TRLs  and  IRLs  are  shown  in  Figure  9. 

The  current  capability  SRL  for  the  Mine-Detection  is  0.622.  According  to  Figure  7,  this 
value  indicates  that  the  capability  is  undergoing  the  Engineering  &  Manufacturing 
Development  phase.  During  this  phase,  the  major  assignments  are  to  develop  system 
capability  or  (increments  thereof),  reduce  integration  and  manufacturing  risk,  ensure 
operational  supportability,  minimize  logistics  footprint,  implement  human  systems 
integration,  design  for  production,  ensure  affordability  and  protection  of  critical 
program  information,  and  demonstrate  system  integration,  interoperability,  safety  and 
utility. 


Figure  9:  Diagram  of  a  System  with  Components  Shaded  for  the  KPP 


Since  we  are  proposing  to  take  into  account  the  resource  consumption  (cost  and  labor- 
hour)  in  the  component  importance  evaluation,  Tables  12  and  13  show  these  values  for 
maturing  the  components  (i.e.,  TRL  and  IRL)  of  the  capability  of  interest.  The  cost  is  in 
thousand  of  dollars  ($1,000),  and  the  effort  is  in  labor-hours.  For  example,  it  requires 
599  hours  of  effort  and  $980,000  to  move  Technology  1  from  level  7  to  level  8.  It  is  the 
obligation  of  the  program  manager  to  obtain  these  estimates  of  resource  consumption  in 
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reality.  To  mature  the  whole  capability,  the  estimated  cost  and  effort  equal  $17,141,000 
and  10,976  of  labor-hours,  respectively. 


Table  12  -  Resource  Consumption  for  TRL  Upgrade 


Tech  1 

Tech  2 

Tech  3 

Tech  4 

Tech  5 

Tech  6 

TRL 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

1 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

2 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

3 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

4 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

5 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

6 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

7 

$0 

0 

$579 

453 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

8 

$980 

599 

$157 

177 

$973 

541 

$459 

154 

$443 

551 

$410 

580 

9 

$820 

290 

$918 

267 

$404 

582 

$592 

341 

$490 

304 

$871 

358 

Sum 

$1,800 

889 

$1,654 

897 

$1,377 

1123 

$1,051 

495 

$933 

855 

$1,281 

938 

Table  13  -  Resource  Consumption  for  IRL  Upgrade 


1,2 

2,3 

2,4 

2,5 

4,5 

5,6 

IRL 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

Cost 

Time 

1 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

2 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

3 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

4 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

5 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

6 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

$0 

0 

7 

$0 

0 

$754 

414 

$968 

509 

$3U 

524 

$0 

$0 

0 

8 

$906 

478 

$382 

90 

$159 

490 

$853 

563 

$613 

392 

$468 

551 

9 

$983 

280 

$735 

220 

$648 

248 

$648 

147 

$374 

370 

$237 

503 

Sum 

$1,889 

758 

$1,871 

724 

$1,775 

1247 

$1,818 

1234 

$987 

762 

$705 

1054 

With  the  proposed  component  Importance  Measures  for  Ip,  ICT,  and  ILH,  this  paper 
considers  two  scenarios  for  each  measure  to  identity  the  importance  of  components 
towards  achieving  the  KPP  in  question.  While  keeping  all  the  other  components 
constant,  the  two  scenarios  are  to  advance  the  current  maturity  of  a  component  to  (1) 
the  next  level,  which  is  trl,  =  trLi  + 1  or  iRLj  =  irLj  + 1 ,  and  (2)  increasing  to  its  highest 

level,  which  is  TRjh=9  or  1  R  7 r  9 . 
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4.4.6  Increasing  Component  Readiness  by  One  Level 

By  increasing  the  component  maturity  by  one  level,  Table  14  shows  the  results  of  the 
calculation.  For  the  Ip  component  importance,  Technology  2  is  the  most  important 
component  whose  change  in  maturity  has  the  largest  impact  on  the  maturity  of  the 
capability.  When  Technology  2  is  increased  by  one  level,  the  Capability  SRL  is  upgraded 
from  its  current  value  of  0.622  to  0.646,  and  gives  an  Ip  of  1.039.  If  the  objective  is  to 
have  the  most  increase  in  Capability  SRL  if  only  one  component  can  be  changed  by  one 
level,  then  Technology  2  is  the  most  important  one.  The  second  and  third  most 
important  components  identified  are  Technologies  5  and  6,  with  an  Ip  of  1.031  and 
1.021,  respectively. 


Table  14  -  Component  Importance  for  the  Scenario  of  Increasing  by  One  Level 


Component 

Current 

Readiness 

Level 

SRL 

P 

P 

Importance 

Rank 

JCT 

JCT 

Importance 

Rank 

JLH 

JLH 

Importance 

Rank 

Technology 

1 

7 

0.634 

1.0195 

5 

1.2E-5 

8 

2.0E-5 

8 

2 

6 

0.646 

1.0390 

1 

4-2E-5 

2 

5-4E-5 

2 

3 

7 

0.634 

1.0189 

6 

1.2E-5 

9 

2.2E-5 

6 

4 

7 

0.634 

1.0197 

4 

2.7E-5 

4 

7-9E-5 

1 

5 

7 

0.641 

1.0307 

2 

4-3E-5 

1 

3-5E-5 

3 

6 

7 

0.635 

1.0207 

3 

3-1E-5 

3 

2.2E-5 

4 

Integration 

1,2 

7 

0.631 

1.0146 

8 

1.0E-5 

11 

1.9E-5 

10 

2,3 

6 

0.631 

1.0146 

9 

1.2E-5 

10 

2.2E-5 

5 

2,4 

6 

0.629 

1.0112 

11 

7.2E-6 

12 

1.4E-5 

11 

2,5 

6 

0.628 

1.0096 

12 

1.9E-5 

6 

1.1E-5 

12 

4,5 

7 

0.631 

1-0135 

10 

1.4E-5 

7 

2.1E-5 

7 

5,6 

7 

0.633 

1.0174 

7 

2.3E-5 

5 

2.0E-5 

9 

For  the  ICT  component  importance,  Technology  5  is  the  most  importance  with  an  ICT  of 
4.3*10-5  indicating  that  the  capability  SRL  will  be  increased  by  4.3*10-5  for  each  dollar 
spent  on  maturing  this  technology.  When  considering  budget  allocation  from  a 
perspective  of  maturing  the  capability,  Technology  5  is  the  most  cost-effective 
component.  The  second  and  third  most  important  components  are  Technologies  2  and 
6. 

Analyzing  the  ILH  component  importance  in  the  same  way,  we  found  that  Technology  4, 
with  an  ILH  of  7.9*10-5  has  the  most  impact  on  capability.  The  capability  SRL  will  be 
upgraded  by  7.9*10-5  for  every  labor-hour  spent  on  maturing  this  technology.  When 
considering  effort  allocation  from  a  perspective  of  maturing  the  capability,  Technology  4 
is  the  most  effort-effective  component.  The  second  and  third  most  important 
components  are  Technologies  2  and  5. 
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Figure  10  puts  together  the  component  importance  evaluation  from  applying  the  three 
IMs  to  the  capability  of  the  system.  The  left  vertical  axis  is  the  scale  for  Ip,  and  the  right 
for  ICT  and  ILH.  Black  bars  represent  the  Ip  importance  with  respect  to  the  importance 
factor  of  TRL/IRL  for  the  corresponding  component,  white  bars  for  the  ICT  importance 
with  respect  to  cost,  and  grey  bar  for  the  ILH  importance  with  respect  to  effort.  The 
higher  the  bar,  the  more  important  is  that  component  with  respect  to  the  importance 
factor  represented  by  the  corresponding  color. 

Therefore,  for  the  scenario  of  increasing  by  one  level,  Technologies  2,  5  and  6  are 
relatively  more  important  than  the  other  components  with  respect  to  TRL/IRL; 
Technologies  5,  2  and  6  are  relatively  more  important  than  others  with  respect  to  cost; 
Technologies  4,  2  and  5  are  relatively  more  important  than  others  with  respect  to  effort. 
When  all  three  importance  factors  are  considered  simultaneously,  Technologies  2,  4  and 
5  are  comparably  more  important  components  for  the  capability  development  within  the 
system.  Furthermore,  Figure  10  implies,  in  general,  that  technologies  are  more 
important  than  integrations  based  on  the  current  development  maturity  status  of  the 
system. 


Increasing  by  One  Level 


1.0*15 
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Figure  10:  Component  Importance  Comparison  for  Increasing  by  One  Level 


4.4.7  Fully  Maturing  Components 

For  the  scenario  of  increasing  the  component  to  its  highest  maturity  level,  Table  15 
shows  the  results  for  considering  each  importance  factor.  Technology  2  is  the  most 
important  component  for  all  three  factors,  indicating  the  significant  impact  of  fully 
maturing  this  technology  on  the  maturity  of  the  capability  of  the  system.  Therefore, 
resources  must  be  prioritized  towards  the  development  of  Technology  2  so  as  to  ensure 
the  satisfaction  of  the  KPP  of  this  system. 
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For  the  consideration  of  importance  factor  of  TRL/IRL,  Technology  5  and  Integration  2, 
3  are  the  second  and  third  most  important  components.  Technology  5  and  Integration  5, 
6  are  the  second  and  third  most  important  with  respect  to  developmental  cost. 
Technologies  4  and  5  are  the  second  and  third  most  important  with  respect  to 
developmental  effort.  It  should  be  noted  here  that  some  integrations  also  stand  as  very 
important  components  for  maturing  the  capability  to  satisfy  the  KPP  of  the  system. 
Again,  results  of  component  importance  calculation  with  respects  to  all  three  factors  are 
plotted  together  in  Figure  11  for  comparison  purpose. 


Table  15  -  Component  Importance  for  the  Scenario  of  Increasing  to  the  Most  Mature  Level 


Component 

Current 

Readiness 

Level 

SRL 

Ip 

Ip 

Importance 

Rank 

JCT 

JCT 

Importance 

Rank 

JLH 

JLH 

Importance 

Rank 

Technology 

1 

7 

0.646 

1.0390 

6 

1-3E-5 

9 

2.7E-5 

6 

2 

6 

0.695 

1.1171 

1 

4-4E-5 

1 

8.1E-5 

1 

3 

7 

0.646 

1.0377 

7 

1-7E-5 

6 

2.1E-5 

9 

4 

7 

0.647 

1.0394 

5 

2.3E-5 

4 

4-9E-5 

2 

5 

7 

0.660 

1.0614 

2 

4.1E-5 

2 

4-5E-5 

3 

6 

7 

0.648 

1.0413 

4 

2.0E-5 

5 

2.7E-5 

5 

Integration 

1,2 

7 

0.640 

1.0291 

10 

9.6E-6 

12 

2.4E-5 

7 

2,3 

6 

0.649 

1.0437 

3 

1-5E-5 

8 

3-8E-5 

4 

2,4 

6 

0.643 

1-0337 

9 

1.2E-5 

10 

1-7E-5 

11 

2,5 

6 

0.640 

1.0288 

11 

9.8E-6 

11 

1-5E-5 

12 

4,5 

7 

0.639 

1.0270 

12 

1-7E-5 

7 

2.2E-5 

8 

5,6 

7 

0.644 

1.0347 

8 

3-1E-5 

3 

2.0E-5 

10 
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Figure  li:  Component  Importance  Comparison  for  Increasing  to  the  Most  Mature  Level 
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5  Appendix  A:  Integration  Readiness  Level 


Below  is  a  series  of  tables  that  contain  a  list  of  decision  criteria  related  to  each  IRL  level. 
It  should  also  be  emphasized  that  the  lists  are  not  considered  to  be  comprehensive  or 
complete;  they  are  merely  an  attempt  to  capture  some  of  the  more  important  decision 
criteria  associated  with  integration  maturity  in  order  to  afford  practitioners  the 
opportunity  to  assess  the  criticality  of  each  decision  criteria  relative  to  the  IRL  it  is  listed 
under.  Thus,  to  establish  further  verification  and  validation  to  the  decision  criteria,  we 
deployed  a  survey  that  asked  Subject  Matter  Experts  (SMEs)  to  evaluate  each  decision 
criteria  in  the  context  of  its  criticality  to  the  specified  IRL.  The  criticality  criteria  for 
assessing  the  IRL  decision  criteria  were  defined  as: 

•  Critical  -  IRL  cannot  be  assessed  without  it 

•  Essential  -  without  it,  IRL  can  be  assessed  but  with  low  to  medium  confidence  in 
the  results 

•  Enhancing  -  without  it,  IRL  can  be  assessed  with  medium  to  high  confidence  in 
the  results 

•  Desirable  -  without  it,  IRL  can  be  assessed  with  very  high  confidence  in  the 
results 

•  N/A  -  the  metric  is  not  applicable  to  the  IRL  assessment 

For  more  details  on  this  study  and  its  results  see  (Sauser  et  al.,  2010). 
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Table  16:  IRL 1  Decision  Criteria  and  Criticality  Assessment 


Relative  Frequency  (RF);  n  =  33 

Cumulative  RF 

IRL  1  Decision  Criteria 

Critical 

Essential 

Enhancing 

Desirable 

N/A 

Critical 

Essential 

Enhancing 

Desirable 

1.1  Principal  integration  technologies  have 

been  identified 

0.58 

0.33 

0.03 

0.06 

0.00 

0.91 

0.09 

1.2  Top-level  functional  architecture  and  interface 

points  have  been  defined 

0.39 

0.52 

0.06 

0.03 

0.00 

0.91 

0.09 

1.3  Availability  of  principal  integration 

technologies  is  known  and  documented 

0.15 

0.39 

0.36 

0.06 

0.03 

0.55 

0.42 

1.4  Integration  concept/plan  has  been 

defined/drafted 

0.18 

0.45 

0.21 

0.12 

0.03 

0.64 

0.33 

1.5  Integration  test  concept/plan  has  been 

defined/drafted 

0.12 

0.36 

0.33 

0.18 

0.00 

0.48 

0.52 

1.6  High-level  Concept  of  Operations  and 

principal  use  cases  have  been  defined/drafted 

0.06 

0.21 

0.55 

0.15 

0.03 

0.27 

0.70 

1.7  Integration  sequence  approach/schedule  has 

been  defined/drafted 

0.06 

0.36 

0.33 

0.21 

0.03 

0.42 

0.55 

1.8  Interface  control  plan  has  been 

defined/drafted 

0.03 

0.12 

0.67 

0.18 

0.00 

0.15 

0.85 

1.9  Principal  integration  and  test  resource 
requirements  (facilities,  hardware,  software, 

surrogates,  etc.)  have  been  defined/identified 

0.09 

0.36 

0.30 

0.18 

0.06 

0.45 

0.48 

1.10  Integration  &  Test  Team  roles  and 
responsibilities  have  been  defined 

0.12 

0.24 

0.33 

0.24 

0.06 

0.36 

0.58 
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Table  17:  IRL  2  Decision  Criteria  and  Criticality  Assessment 


Relative  Frequency  (RF);  n  =  33 

Cumulative  RF 

IRL  2  Decision  Criteria 

Critical 

Essential 

Enhancing 

Desirable 

N/A 

Critical 

Essential 

Enhancing 

Desirable 

2.1  Principal  integration  technologies  function  as 

stand-alone  units 

0.18 

0.27 

0.24 

0.30 

0.00 

0.45 

0.55 

2.2  Inputs/outputs  for  principal  integration 
technologies  are  known,  characterized  and 

documented 

0.52 

0.36 

0.06 

0.06 

0.00 

0.88 

0.12 

2.3  Principal  interface  requirements  for  integration 

technologies  have  been  defined/drafted 

0.39 

0.33 

0.24 

0.03 

0.00 

0.73 

0.27 

2.4  Principal  interface  requirements  specifications 
for  integration  technologies  have  been 

defined/drafted 

0.27 

0.45 

0.24 

0.03 

0.00 

0.73 

0.27 

2.5  Principal  interface  risks  for  integration 
technologies  have  been  defined/drafted 

0.06 

0.24 

0.61 

0.09 

0.00 

0.30 

0.70 

2.6  Integration  concept/plan  has  been  updated 

0.06 

0.42 

0.42 

0.09 

0.00 

0.48 

0.52 

2.7  Integration  test  concept/plan  has  been 

updated 

0.09 

0.27 

0.52 

0.12 

0.00 

0.36 

0.64 

2.8  High-level  Concept  of  Operations  and 
principal  use  cases  have  been  updated 

0.12 

0.18 

0.45 

0.21 

0.03 

0.30 

0.67 

2.9  Integration  sequence  approach/schedule  has 

been  updated 

0.09 

0.27 

0.45 

0.18 

0.00 

0.36 

0.64 

2.10  Interface  control  plan  has  been  updated 

0.06 

0.30 

0.61 

0.03 

0.00 

0.36 

0.64 

2.11  Integration  and  test  resource  requirements 
(facilities,  hardware,  software,  surrogates,  etc.) 

have  been  updated 

0.15 

0.39 

0.27 

0.15 

0.03 

0.55 

0.42 

2.12  Long  lead  planning/coordination  of 
integration  and  test  resources  have  been  initiated 

0.12 

0.30 

0.30 

0.24 

0.03 

0.42 

0.55 

2.13  Integration  &  Test  Team  roles  and 

responsibilities  have  been  updated 

0.03 

0.15 

0.58 

0.21 

0.03 

0.18 

0.79 

2.14  Formal  integration  studies  have  been 

initiated 

0.12 

0.33 

0.21 

0.21 

0.12 

0.45 

0.42 
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Table  18:  IRL  3  Decision  Criteria  and  Criticality  Assessment 


Relative  Frequency  (RF);  n  =  33 

Cumulative  RF 

IRL  3  Decision  Criteria 

Critical 

Essential 

Enhancing 

Desirable 

N/A 

Critical 

Essential 

Enhancing 

Desirable 

3.1  Preliminary  Modeling  &  Simulation  and/or 
analytical  studies  have  been  conducted  to  identify 
risks  &  assess  compatibility  of  integration 

technologies 

0.18 

0.36 

0.45 

0.00 

0.00 

0.55 

0.45 

3.2  Compatibility  risks  and  associated  mitigation 

strategies  for  integration  technologies  have  been 

defined  (initial  draft) 

0.09 

0.39 

0.52 

0.00 

0.00 

0.48 

0.52 

3.3  Integration  test  requirements  have  been 

defined  (initial  draft) 

0.15 

0.48 

0.24 

0.12 

0.00 

0.64 

0.36 

3.4  High-level  system  interface  diagrams  have 

been  completed 

0.48 

0.27 

0.24 

0.00 

0.00 

0.76 

0.24 

3.5  Interface  requirements  are  defined  at  the 

concept  level 

0.24 

0.70 

0.06 

0.00 

0.00 

0.94 

0.06 

3.6  Inventory  of  external  interfaces  is  completed 

0.24 

0.33 

0.42 

0.00 

0.00 

0.58 

0.42 

3.7  Data  engineering  units  are  identified  and 

documented 

0.06 

0.45 

0.24 

0.21 

0.03 

0.52 

0.45 

3.8  Integration  concept  and  other  planning 

documents  have  been  modified/updated  based  on 

preliminary  analyses 

0.18 

0.27 

0.42 

0.09 

0.03 

0.45 

0.52 
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Table  19:  IRL  4  Decision  Criteria  and  Criticality  Assessment 


Relative  Frequency  (RF);  n  =  33 

Cumulative  RF 

IRL  4  Decision  Criteria 

Critical 

Essential 

Enhancing 

Desirable 

N/A 

Critical 

Essential 

Enhancing 

Desirable 

4.1  Quality  Assurance  plan  has  been  completed 

and  implemented 

0.18 

0.27 

0.36 

0.15 

0.03 

0.45 

0.52 

4.2  Cross  technology  risks  have  been  fully 

identified/characterized 

0.12 

0.52 

0.33 

0.03 

0.00 

0.64 

0.36 

4.3  Modeling  &  Simulation  has  been  used  to 

simulate  some  interfaces  between  components 

0.06 

0.24 

0.70 

0.00 

0.00 

0.30 

0.70 

4.4  Formal  system  architecture  development  is 

beginning  to  mature 

0.09 

0.52 

0.36 

0.03 

0.00 

0.61 

0.39 

4.5  Overall  system  requirements  for  end 

users’  application  are  known/baselined 

0.24 

0.55 

0.15 

0.06 

0.00 

0.79 

0.21 

4.6  Systems  Integration  Laboratory/Software  test¬ 
bed  tests  using  available  integration  technologies 

have  been  completed  with  favorable  outcomes 

0.09 

0.52 

0.36 

0.03 

0.00 

0.61 

0.39 

4.7  Low  fidelity  technology  “system”  integration 
and  engineering  has  been  completed  and  tested 

in  a  lab  environment 

0.06 

0.36 

0.52 

0.06 

0.00 

0.42 

0.58 

4.8  Concept  of  Operations,  use  cases  and 

Integration  requirements  are  completely  defined 

0.12 

0.30 

0.55 

0.00 

0.03 

0.42 

0.55 

4.9  Analysis  of  internal  interface  requirements  is 
completed 

0.09 

0.61 

0.27 

0.03 

0.00 

0.70 

0.30 

4.10  Data  transport  method(s)  and  specifications 

have  been  defined 

0.12 

0.36 

0.48 

0.03 

0.00 

0.48 

0.52 

4.11  A  rigorous  requirements  inspection  process 

has  been  implemented 

0.27 

0.30 

0.21 

0.21 

0.00 

0.58 

0.42 
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Table  20:  IRL  5  Decision  Criteria  and  Criticality  Assessment 


Relative  Frequency  (RF);  n  =  33 

Cumulative  RF 

IRL  5  Decision  Criteria 

Critical 

Essential 

Enhancing 

Desirable 

N/A 

Critical 

Essential 

Enhancing 

Desirable 

5.1  An  Interface  Control  Plan  has  been 

implemented  (i.e.,  Interface  Control  Document 
created,  Interface  Control  Working  Group 
formed,  etc.) 

0.33 

0.58 

0.06 

0.00 

0.03 

0.91 

0.06 

5.2  Integration  risk  assessments  are  ongoing 

0.06 

0.48 

0.45 

0.00 

0.00 

0.55 

0.45 

5.3  Integration  risk  mitigation  strategies  are  being 

implemented  &  risks  retired 

0.03 

0.52 

0.39 

0.06 

0.00 

0.55 

0.45 

5.4  System  interface  requirements  specification 

has  been  drafted 

0.39 

0.36 

0.24 

0.00 

0.00 

0.76 

0.24 

5.5  External  interfaces  are  well  defined  (e.g., 

source,  data  formats,  structure,  content,  method 

of  support,  etc.) 

0.27 

0.55 

0.18 

0.00 

0.00 

0.82 

0.18 

5.6  Functionality  of  integrated  configuration  items 
(modules/functions/assemblies)  has  been 
successfully  demonstrated  in  a 

laboratory/synthetic  environment 

0.21 

0.52 

0.27 

0.00 

0.00 

0.73 

0.27 

5.7  The  Systems  Engineering  Management  Plan 
addresses  integration  and  the  associated 

interfaces 

0.15 

0.18 

0.33 

0.12 

0.21 

0.33 

0.45 

5.8  Integration  test  metrics  for  end-to-end  testing 

have  been  defined 

0.12 

0.33 

0.52 

0.03 

0.00 

0.45 

0.55 

5.9  Integration  technology  data  has  been 

successfully  modeled  and  simulation 

0.06 

0.67 

0.18 

0.09 

0.00 

0.73 

0.27 
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Table  21:  IRL  6  Decision  Criteria  and  Criticality  Assessment 


Relative  Frequency  (RF);  n  =  33 

Cumulative  RF 

IRL  6  Decision  Criteria 

Critical 

Essential 

Enhancing 

Desirable 

N/A 

Critical 

Essential 

Enhancing 

Desirable 

6.1  Cross  technology  issue  measurement  and 

performance  characteristic  validations  completed 

0.27 

0.39 

0.33 

0.00 

0.00 

0.67 

0.33 

6.2  Software  components  (operating  system, 
middleware,  applications)  loaded  onto 

subassemblies 

0.45 

0.33 

0.12 

0.03 

0.06 

0.79 

0.15 

6.3  Individual  modules  tested  to  verify  that  the 
module  components  (functions)  work  together 

0.48 

0.42 

0.09 

0.00 

0.00 

0.91 

0.09 

6.4  Interface  control  process  and  document  have 

stabilized 

0.09 

0.48 

0.36 

0.03 

0.03 

0.58 

0.39 

6.5  Integrated  system  demonstrations  have  been 

successfully  completed 

0.21 

0.58 

0.15 

0.06 

0.00 

0.79 

0.21 

6.6  Logistics  systems  are  in  place  to  support 
Integration 

0.12 

0.42 

0.27 

0.18 

0.00 

0.55 

0.45 

6.7  Test  environment  readiness  assessment 

completed  successfully 

0.06 

0.52 

0.33 

0.06 

0.03 

0.58 

0.39 

6.8  Data  transmission  tests  completed 
successfully 

0.18 

0.64 

0.06 

0.06 

0.06 

0.82 

0.12 

Table  22:  IRL  7  Decision  Criteria  and  Criticality  Assessment 


Relative  Frequency  (RF);  n  =  33 

Cumulative  RF 

IRL  7  Decision  Criteria 

Critical 

Essential 

Enhancing 

Desirable 

N/A 

Critical 

Essential 

Enhancing 

Desirable 

7.1  End-to-end  Functionality  of  Systems 
Integration  has  been  successfully 

demonstrated 

0.61 

0.18 

0.21 

0.00 

0.00 

0.79 

0.21 

7.2  Each  system/software  interface  tested 

individually  under  stressed  and  anomalous 

conditions 

0.33 

0.55 

0.12 

0.00 

0.00 

0.88 

0.12 

7.3  Fully  integrated  prototype  demonstrated  in 

actual  or  simulated  operational  environment 

0.42 

0.45 

0.09 

0.03 

0.00 

0.88 

0.12 

7.4  Information  control  data  content  verified  in 

system 

0.24 

0.55 

0.18 

0.00 

0.03 

0.79 

0.18 

7.5  Interface,  Data,  and  Functional  Verification 

0.33 

0.55 

0.09 

0.03 

0.00 

0.88 

0.12 

7.6  Corrective  actions  planned  and  implemented 

0.15 

0.48 

0.27 

0.09 

0.00 

0.64 

0.36 
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Table  23:  IRL  8  Decision  Criteria  and  Criticality  Assessment 


Relative  Frequency  (RF);  n  =  33 

Cumulative  RF 

IRL  8  Decision  Criteria 

Critical 

Essential 

Enhancing 

Desirable 

N/A 

Critical 

Essential 

Enhancing 

Desirable 

8.1  All  integrated  systems  able  to  meet  overall 
system  requirements  in  an  operational 

environment 

0.85 

0.12 

0.03 

0.00 

0.00 

0.97 

0.03 

8.2  System  interfaces  qualified  and  functioning 
correctly  in  an  operational  environment 

0.61 

0.36 

0.03 

0.00 

0.00 

0.97 

0.03 

8.3  Integration  testing  closed  out  with  test  results, 

anomalies,  deficiencies,  and  corrective  actions 

documented 

0.39 

0.52 

0.09 

0.00 

0.00 

0.91 

0.09 

8.4  Components  are  form,  fit,  and  function 

compatible  with  operational  system 

0.42 

0.48 

0.06 

0.03 

0.00 

0.91 

0.09 

8.5  System  is  form,  fit,  and  function  design  for 

intended  application  and  operational  environment 

0.42 

0.45 

0.09 

0.03 

0.00 

0.88 

0.12 

8.6  Interface  control  process  has  been 

completed/closed-out 

0.24 

0.45 

0.24 

0.06 

0.00 

0.70 

0.30 

8.7  Final  architecture  diagrams  have  been 

submitted 

0.36 

0.12 

0.42 

0.09 

0.00 

0.48 

0.52 

8.8  Effectiveness  of  corrective  actions  taken  to 

close-out  principal  design  requirements  has  been 

demonstrated 

0.24 

0.48 

0.24 

0.03 

0.00 

0.73 

0.27 

8.9  Data  transmission  errors  are  known, 

characterized  and  recorded 

0.36 

0.33 

0.21 

0.09 

0.00 

0.70 

0.30 

8.10  Data  links  are  being  effectively  managed  and 

process  improvements  have  been  initiated 

0.18 

0.52 

0.27 

0.03 

0.00 

0.70 

0.30 

Table  16:  IRL  9  Decision  Criteria  and  Criticality  Assessment 


Relative  Frequency  (RF);  n  =  33 

Cumulative  RF 

IRL  9  Decision  Criteria 

Critical 

Essential 

Enhancing 

Desirable 

N/A 

Critical 

Essential 

Enhancing 

Desirable 

9.1  Fully  integrated  system  has  demonstrated 
operational  effectiveness  and  suitability  in  its 

intended  or  a  representative  operational 

environment 

0.82 

0.09 

0.09 

0.00 

0.00 

0.91 

0.09 

9.2  Interface  failures/failure  rates  have  been  fully 

characterized  and  are  consistent  with  user 

requirements 

0.64 

0.27 

0.06 

0.03 

0.00 

0.91 

0.09 

9.3  Lifecycle  costs  are  consistent  with  user 
requirements  and  lifecycle  cost  improvement 

initiatives  have  been  initiated 

0.24 

0.42 

0.21 

0.09 

0.03 

0.67 

0.30 
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